Introducing tcprstat, a TCP response time tool

Ignacio Nin and I (mostly Ignacio) have worked together to create tcprstat[1], a new tool that times TCP requests and prints out statistics on them. The output looks somewhat like vmstat or iostat, but we’ve chosen the statistics carefully so you can compute meaningful things about your TCP traffic.

What is this good for? In a nutshell, it is a lightweight way to measure response times on a server such as a database, memcached, Apache, and so on. You can use this information for historical metrics, capacity planning, troubleshooting, and monitoring to name just a few.

The tcprstat tool itself is a means of gathering raw statistics, which are suitable for storing and manipulating with other programs and scripts. By default, tcprstat works just like vmstat: it runs once, prints out a line, and exits. You’ll probably want to tell it to run forever, and continue to print out more lines. Each line contains a timestamp and information about the response time of the requests within that time period. Here “response time” means, for a given TCP connection, the time elapsed from the last inbound packet until the first outbound packet. For many simple protocols such as HTTP and MySQL, this is the moral equivalent of a query’s response time.

The statistics we chose to output by default are the count, median, average, min, max, and standard deviation of the response times, in microseconds. These are repeated for the 95th and 99th percentiles as well. Other metrics are also available. Here’s a sample:

[root@server] # tcprstat -p 3306 -n 0 -t 1
timestamp	count	max	min	avg	med	stddev	95_max	95_avg	95_std	99_max	99_avg	99_std
1276827985	1341	24556	23	149	59	767	310	91	69	1030	107	112
1276827986	1329	12098	28	134	63	461	299	91	65	667	104	93
1276827987	1180	13277	22	202	93	873	439	103	79	1523	131	169
1276827988	1441	15878	27	180	139	672	427	116	79	1045	136	128
1276827989	1432	157198	26	272	138	4165	405	115	80	1092	134	123
1276827990	1835	25198	26	183	124	734	448	115	85	1141	137	141
1276827991	1242	6949	29	129	114	301	233	98	61	686	109	84
1276827992	1480	284181	25	442	127	7432	701	128	114	4157	173	293
1276827993	1448	9339	22	161	88	425	392	104	80	1280	126	140

tcprstat uses libpcap to capture traffic. It’s a threaded application that does the minimum possible work and uses efficient data structures. Your feedback on the kernel/userland exchange overhead caused by the packet sniffing would be very appreciated — libpcap allows the user to tune this exchange, so if you have suggestions on how to improve it, that’s great.

We build statically linked binaries with the preferred version of libpcap, which means there are no dependencies. You can just run the tool. In the future, packages in the Percona repositories will provide another means for rapid installation via yum and apt.

tcprstat is beta software. Several C/C++ experts reviewed its code and gave it a thumbs-up, so many eyes have been on the code. We’ve performed tests on servers with high loads and observed minimal resource consumption. I personally have been running it for many weeks on some production servers without stopping it and have seen no problems, so I am pretty sure it has no memory leaks or other problems. Nevertheless, it’s a first prototype release, and we want much more testing. We might also change the functionality; as we build tools around it, we discover new things that might be useful. When we’re happy with it and you’re happy with it, we’ll take the Beta label away and make it GA.

The tcprstat user’s manual and links to downloads are on the Percona wiki. Commercial support and services are provided by Percona. Bug reports, feature requests, etc should go to the Launchpad project linked from the user’s manual. General discussion is welcome on the Google Group also linked from the user’s manual.

[1] Historical note: we initially called this tool rtime, but did not publicize it. However, some of you might have heard of “rtime” before. This is the same tool.

Entry posted by Baron Schwartz |

Add to: delicious | digg | reddit | netscape | Google Bookmarks


InnoDB memory allocation, ulimit, and OpenSUSE

I recently encountered an interesting case. A customer reported that mysqld crashed on start on OpenSUSE 11.2 kernel x86_64   with 96 GB RAM when the innodb_buffer_pool_size was set to anything more than 62 GB. I decided to try it with 76 GB. The error message was an assert due to a failed malloc() in ut_malloc_low() in ut/ut0mem.c inside InnoDB source code. InnoDB wraps the majority of its memory allocations in ut_malloc_low(), so to get an idea of the pattern of requested allocations I added a debugging fprintf() to tell me how much was being allocated and whether it was successful.

I discovered something interesting. I expected the allocation to fail on the 76 GB of the buffer pool, due to some weird memory mapping issue and a continuous block of 76 GB not being available. However, that is not what happened. 76 GB buffer was allocated successfully. What was failing is the allocation of 3.37GB after that. What in the world could InnoDB need that was 3.37 GB? There was nothing in the settings that asked for anything close to 3 GB explicitly.

Source code is the ultimate documentation, and I took advantage of that. My good friend GDB guided me to buf_pool_init() in buf/buf0buf.c. There I found the following:

buf_pool->frame_mem = os_mem_alloc_large(
UNIV_PAGE_SIZE * (n_frames + 1),

That was the buffer pool itself, the 76 GB of it. And now the buffer pool’s friend:

buf_pool->blocks = ut_malloc(sizeof(buf_block_t) * max_size);

3.6 GB of it!

From the comments in the code (InnoDB code actually has very good comments), max_size is the maximum number of buffer pool pages (16K each), n_frames which is the same thing unless AWE is used, but it was not used, so I did not worry about it.

What shall we call that friend? It is used for storing some meta information about buffer pool pages. The most natural name I could come up with from reading the source code is the blocks array.

Thus we can see that we are allocating another chunk that is in proportion to the setting of innodb_buffer_pool_size for the blocks array. The exact proportions will probably vary from version to version, but roughly about 1 G for every 25 G of the buffer pool. This can become significant in the proper innodb_buffer_pool_size estimations when the system has a lot of RAM and you want to have the largest possible innodb_buffer_pool_size. Do not forget to give the blocks array some room!

While this was an interesting investigation, it nevertheless did not explain why there was not enough room for a 76 GB buffer pool. Even with the extra 3.37 GB allocation, there was still some free memory. Or was there? Maybe some hidden monster was eating it up? I quickly wrote this hack to prove or disprove the monster’s presence.

I verified that I could allocate and initialize two chunks of 40 GB from two separate processes, but not 80 GB from one. In fact, 80GB allocation failed right in malloc(), did not even get to initialization. I tested it with allocating 70 GB concurrently in each process so as to overrun physical memory + swap. Both allocations were successful, one initialized successfully, the other was killed by the OOM kill during initialization.

This smelled like a low ulimit, and sure enough it was. ulimit -m ulimited; ulimit -v unlimited did the magic, and mysqld successfully started with an 80 GB buffer pool. Apparenly OpenSUSE defaults are set in proportion to physical memory to keep the memory-hungry applications from taking the system down. On this particular system (96 GB physical memory, 2 GB swap it decided to set the virtual memory ulimit (-v) to 77.27 GB, and the physical memory (-m) to 80.40 GB).

Entry posted by Sasha Pachev |

Add to: delicious | digg | reddit | netscape | Google Bookmarks

Written by in: MySQL,Zend Developer |

Prison for water..

Punishment for early escape: You have to help destroy Los Angeles

Conditions for release on time: you have to help destroy Los Angeles


High availability for MySQL on Amazon EC2 – Part 4 – The instance restart script

This post is the fourth of a series that started here.

From the previous of this series, we now have resources configured but instead of starting MySQL, Pacemaker invokes a script to start (or restart) the EC2 instance running MySQL. This blog post describes the instance restart script. Remember, I am more a DBA than a script writer so it might not be written in the most optimal way.

First, let’s recap what’s the script has to perform (the full script is given below).

  1. Kill the MySQL EC2 instance if running
  2. Make sure the MySQL EC2 instance is stopped
  3. Prepare the user-data script for the new MySQL EC2 instance
  4. Launch the new MySQL instance
  5. Make sure it is running
  6. Reconfigure local heartbeat
  7. Broadcast the new MySQL instance IP to the application servers

Kill the MySQL EC2 instance

In order to kill the existing MySQL EC2 instance, we first have to identify it. This is done by:


  1. OLD_INSTANCE_ID=`ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/ | grep $AMI_HA_MYSQL | egrep “running|pending” | tail -n 1 | cut -d‘|’ -f3`

by filtering on the AMI type of the instance. Since an instance can be listed at the “stopped” state, it is mandatory to filter for states “running” or “pending”. Then the instance is terminated with:


  1. ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null

Make sure the MySQL EC2 instance is stopped

Terminating an EC2 instance is not instantaneous, we can confirm an instance is really stopped by monitoring its status and wait until it is actually “terminated”. The code below is how the script performs this task.


  1. #wait until the old instance is terminated  it takes a few seconds to stop
  2. done=“false”
  3. while [ $done == “false” ]
  4. do
  5.     status=`ec2-describe-instances -K $PK -C $CERT $OLD_INSTANCE_ID | /usr/local/bin/ |  grep -c terminated`
  6.       if [ “$status” -eq “1” ]; then
  7.             done=“true”
  8.         else
  9.                 ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
  10.             sleep 5
  11.      fi
  12. done

Prepare the user-data script for the new MySQL EC2 instance

The new MySQL instance will be running heartbeat. Since we cannot use neither Ethernet broadcast or multicast, we need to configure the new instance so that it communicates through unicast with its partner node in the cluster, the node on which the restart script is run. This configuration is achieved by providing a user-data script (see the hamysql.user-data below) which completes the heartbeat configuration of the new instance. The hamysql.user-data script only performs a search and replace operation on the /etc/ha.d/ file and then restart the heartbeat service. In order for this to work properly, we just have to put the IP of the current instance in the script like here:


  1. OUR_IP=`/sbin/ifconfig eth0 | grep ‘inet addr’ | cut -d‘:’ -f2 | cut -d‘ ‘ -f1`
  2. #Now, modify the user-data script, we need to put our IP address in
  3. if [ “$OUR_IP” == “” ]
  4. then
  5.     echo “Error getting Our IP”
  6. else   
  7.     perl -pi -e “s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $OUR_IP/g” $USER_DATA_SCRIPT
  8. fi

Launch the new MySQL instance

Once things are ready, a new MySQL instance can be launched with:


  1. #Now we are ready to start a new one
  4. #wait until the new instance is running  it take a few seconds to start
  5. NEW_INSTANCE_ID=`echo $INSTANCE_INFO | cut -d‘|’ -f3`

Out of this operation, we retrieve the new instance “instance_id”.

Make sure it is running

Since we know the “instance_id” of the new instance, checking if it is running is easy:


  1. done=“false”
  2. while [ $done == “false” ]
  3. do
  4.     INSTANCE_INFO=`ec2-describe-instances -K $PK -C $CERT  $NEW_INSTANCE_ID | /usr/local/bin/`
  5.     status=`echo $INSTANCE_INFO | grep -c running`
  6.     if [ “$status” -eq “1” ]; then
  7.         done=“true”
  8.     else
  9.         sleep 5
  10.     fi
  11. done

Reconfigure local heartbeat

Now, Heartbeat, on the monitoring host, must be informed of the IP address of its new partner. In order to achieve this, a search and replace operation in the local file followed of restart of Heartbeat is sufficient.


  1. #Set the IP in /etc/ha.d/ and ask heartbeat to reload its config
  2. MYSQL_IP=`ec2-describe-instances -K $PK -C $CERT  $NEW_INSTANCE_ID | /usr/local/bin/ | cut -d‘|’ -f2`
  3. perl -pi -e “s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $MYSQL_IP/g” /etc/ha.d/
  4. /etc/init.d/heartbeat reload

Broadcast the new MySQL instance IP to the application servers

The final phase is to inform the application servers that the IP of the MySQL has changed. The best way to list those application servers is through a security group and, provided the appropriate ssh keys have been exchanged, this code will push the IP update.


  1. TMPFILE=`mktemp`
  2. ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/ | grep $CLIENT_SECURITY_GROUP> $TMPFILE
  4. while read line
  5. do
  6.     IP=`echo $line | cut -d‘|’ -f2`
  7.     ssh -i /usr/local/bin/update_mysql ubuntu@$IP sudo ./ $MYSQL_IP
  8. done <$TMPFILE
  10. rm $TMPFILE

The full script:


  1. #!/bin/bash
  2. HA_SECURITY_GROUP=testyves
  3. CLIENT_SECURITY_GROUP=hamysql-client
  4. CLIENT_SCRIPT=/usr/local/bin/
  5. AMI_HA_MYSQL=ami-84a74fed
  6. EBS_DATA_VOL=vol-aefawf
  7. USER_DATA_SCRIPT=/usr/local/bin/hamysql.user-data
  10. INSTANCE_TYPE=m1.small
  11. INSTANCE_ZONE=us-east-1c
  12. INSTANCE_KEY=yves-key
  14. OUR_IP=`/sbin/ifconfig eth0 | grep ‘inet addr’ | cut -d‘:’ -f2 | cut -d‘ ‘ -f1`
  15. #Now, modify the user-data script, we need to put our IP address in
  16. if [ “$OUR_IP” == “” ]
  17. then
  18.     echo “Error getting Our IP”
  19. else   
  20.     perl -pi -e “s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $OUR_IP/g” $USER_DATA_SCRIPT
  21. fi
  23. while [ 1 ]; do
  25.     #First thing to do, terminate the other instance ID
  26.     OLD_INSTANCE_ID=`ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/ | grep $AMI_HA_MYSQL | egrep “running|pending” | tail -n 1 | cut -d‘|’ -f3`
  28.     if [ “$OLD_INSTANCE_ID” == “” ]
  29.     then
  30.         #no running instance
  31.         :
  32.     else
  33.         ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
  35.         #wait until the old instance is terminated  it takes a few seconds to stop
  36.         done=“false”
  37.         while [ $done == “false” ]
  38.         do
  39.             status=`ec2-describe-instances -K $PK -C $CERT $OLD_INSTANCE_ID | /usr/local/bin/ |  grep -c terminated`
  40.          if [ “$status” -eq “1” ]; then
  41.               done=“true”
  42.       else
  43.                 ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
  44.                 sleep 5
  45.       fi
  46.         done   
  47.     fi
  49.     #Now we are ready to start a new one
  50.     INSTANCE_INFO=`ec2-run-instances -K $PK -C $CERT $AMI_HA_MYSQL -n 1 -g $HA_SECURITY_GROUP -f $USER_DATA_SCRIPT -t $INSTANCE_TYPE -z $INSTANCE_ZONE -k $INSTANCE_KEY | /usr/local/bin/`
  52.     #wait until the new instance is running  it take a few seconds to start
  53.     NEW_INSTANCE_ID=`echo $INSTANCE_INFO | cut -d‘|’ -f3`
  55.     if [ “$NEW_INSTANCE_ID” == “” ]
  56.     then
  57.         echo “Error creating the new instance”
  58.     else
  60.         done=“false”
  61.         while [ $done == “false” ]
  62.         do
  63.       INSTANCE_INFO=`ec2-describe-instances -K $PK -C $CERT  $NEW_INSTANCE_ID | /usr/local/bin/`
  64.          status=`echo $INSTANCE_INFO | grep -c running`
  65.       if [ “$status” -eq “1” ]; then
  66.               done=“true”
  67.          else
  68.                 sleep 5
  69.       fi
  70.         done
  72.         #Set the IP in /etc/ha.d/ and ask heartbeat to reload its config
  73.         MYSQL_IP=`ec2-describe-instances -K $PK -C $CERT  $NEW_INSTANCE_ID | /usr/local/bin/ | cut -d‘|’ -f2`
  74.         perl -pi -e “s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $MYSQL_IP/g” /etc/ha.d/
  76.         TMPFILE=`mktemp`
  77.         ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/ | grep $CLIENT_SECURITY_GROUP> $TMPFILE
  79.         while read line
  80.         do
  81.             IP=`echo $line | cut -d‘|’ -f2`
  82.             ssh -i /usr/local/bin/update_mysql ubuntu@$IP sudo ./ $MYSQL_IP
  83.         done <$TMPFILE
  85.         rm $TMPFILE
  87.         /etc/init.d/heartbeat reload
  88.     fi
  91.         sleep 300 # 5 min before attempting again. Normally heartbeat should kill the script before
  92. done

The hamysql.user-data script:

The script sets the IP of the monitor host in the heartbeat configuration file and then, finishes up some missing configuration settings of the AMI.


  1. #!/bin/bash
  2. sudo hostname hamysql
  3. sudo perl -pi -e “s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0” /etc/ha.d/
  5. # to eventually be added to the ebs image
  6. sudo perl -pi -e ‘s/bind-address/#bind-address/g’ /etc/mysql/my.cnf
  7. sudo service mysql restart
  8. sleep 5
  9. /usr/bin/mysql -u root -proot -e “grant all on *.* to root@’%’ identified by ‘root'”
  10. sudo /etc/init.d/heartbeat start

Entry posted by Yves Trudeau |
No comment

Add to: delicious | digg | reddit | netscape | Google Bookmarks


Percona talks at OpenSQL Camp this weekend

Four Perconians (perconites?) will be at OpenSQL Camp in Sankt Augustin, Germany this weekend presenting talks on:

  • Recovery of Lost or Corrupted InnoDB Tables
  • Keep your MySQL backend online no matter what
  • XtraDB — InnoDB on steroids
  • Xtrabackup for MySQL

If you would like to stop by and say hello, we are Aleksandr, Istvan, Morgan and Aurimas (pictures here).

If you can make the (approximate) location, but not the date, we also have training in Frankfurt in three weeks time.

Entry posted by Morgan Tocker |

Add to: delicious | digg | reddit | netscape | Google Bookmarks


Announcing Training for Operations Teams

We’re opening up registration for our new training courses today.  In short: we are moving from two days to a new four-day format.  The new additions are created by:

  • Splitting our current InnoDB day in half. We now have one day for DBAs, and one day just on InnoDB topics.
  • A new Operations Day – covering how to maintain MySQL in production.

Our developer course has also undergone revision, and we now have more query tuning examples, and a new instrumentation chapter.

What is operations training?

Many companies split their technical staff between development, and operations.  The operations team is responsible for tasks such as capacity planning, backup/disaster recovery, and carrying a pager. They are the heroes that fight fires.

Our operations day of training is delivered as a hands on class.  Attendees will be divided into teams, and given a series of challenges to complete on EC2 machines.  As part of the development of this course we wrote a sample LAMP application, embedded with minor flaws which students will need to fix while they try and keep the application up.

Where can you attend?

We’re starting off by branding the operations day as a ‘BETA’.  You can attend for only $100 at San Francisco Thursday, 30 Sep 2010, New York Thursday, 14 Oct 2010 or Vancouver Thursday, 21 Oct 2010. There is also a discount of attending all four days for $1,450 if you book before 30 August.

After our initial BETA, the courses will be available in more USA and international locations. A partial list is already available on the training section of our website. We will confirm more cities in the coming weeks.

Entry posted by Morgan Tocker |

Add to: delicious | digg | reddit | netscape | Google Bookmarks


Ubuntu 10.04.1 LTS released | The Fridge

Ubuntu 10.04.1 LTS released | The Fridge.

This is pretty cool. I know as a system administrator, I never wanted to run .0 anything. So 10.04 is really like 10.04.0, and means “let somebodye lse find the bugs.”.

Well 10.04.1 means that the more conservative administrators can at least have a reasonable expectation that it will be even more stable than it was on release day in April.

If you’re already running Ubuntu servers, btw, check this out:

Hit it and be counted as a server user. Pretty amazing how many little orange circles there are all over the world.

For those of you who’ve been telling me that my blog posts sound like “gleep ork boog florg”, a quick primer:

Ubuntu is an operating system, like Mac OS X or Windows (except more awesomer).

10.04 was their April, 2010 release (10 == 2010 04 == april).

LTS means Long Term Support. This means that the people who maintain Ubuntu will support this release for 3 – 5 years (depending on the context.. 3 for desktops, 5 for servers).

10.04.1 is a fixed up release, mainly marking the release of updated CD images for installing. If you install 10.04 and choose automatic updates, you’re already on 10.04.1 before the release.


Testing MySQL column stores

Recently I had the opportunity to do some testing on a large data set against two MySQL column-store storage engines. I’d like to note that this effort was sponsored by Infobright, but this analysis reflects my independent testing from an objective viewpoint.

I performed two different types of testing. The first focused on core functionality and compatibility of ICE (Infobright Community Edition) compared with MyISAM on a small data set. The second part of my testing compared the performance and accuracy of ICE with InfiniDB Community Edition on a 950GB data set.

The first first part of my analysis focused on testing specific MySQL functionally with Infobright’s storage engine. A lot of my tests involved corner or edge cases since I have experience developing such cases for MySQL column-based storage engines in the past. I reported any bugs that I found, and contributed my test cases to ICE. In fact, some of the issues have already been addressed in the most recent software release. An example of such a problem would be “select avg(char_column) from table” where the column contains a mix of ascii data, such as names and numeric data. This is an example of implicit casting that probably wouldn’t happen in a real application.

Importantly, I didn’t find significant defects in Infobright that would be “show stoppers” for typical OLAP analysis queries. These tests were intended to approximate what would happen if you ported a MyISAM OLAP application to ICE. My testing suggests that with some basic testing, an application could be ported to ICE with a good chance of success. Of course, a good test environment is something that I think every operations team should insist on.

The second part of my analysis focused on testing a total of 29 queries on the large data set. I compared a number of different factors between the two databases, including:

  1. Ease of installation
  2. Loading capability and speed of loading
  3. Accuracy of results of queries over the large data set
  4. Speed of results of queries over the large data set
  5. Basic security aspects

The report that I produced may be found here.

It should be noted that the second set of tests included 29 different queries, some of which were provided by Infobright and others which I contributed. InfiniDB does not support as many data types and aggregate functions as ICE, and therefore it could not run some of the queries. ICE supports almost all of the MySQL aggregation functions. Notably, GROUP_CONCAT is not supported, which is something I hope they rectify in a future version.

In addition, I was not able to get accurate results for all of the queries on InfiniDB. In particular the query “select count(*) from carsales.sales_fact” when run on InfiniDB returned a number that was higher than it should have been, and several GROUP BY queries returned unexpected results as well. I did not change any “out of the box” settings for Infobright. Even after I modified the configuration settings on a 16GB box, one query did not have enough memory to complete on InfiniDB.

Overall, I would say that ICE is more “ready for prime time” than InfiniDB given the inconsistencies that I encountered. I will try to reproduce the problems on InfiniDB into easily reproducible test cases which I may contribute to them, but this is difficult given the size of the data set involved. ICE was able to execute queries quickly, and with accurate results.

Entry posted by Justin Swanhart |

Add to: delicious | digg | reddit | netscape | Google Bookmarks


Ubuntu Bug Day!

If you love Ubuntu, and want to help out, join us for Ubuntu Bug Day tomorrow!

Written by in: Ubuntu,Zend Developer |

Why message queues and offline processing are so important

If you read Percona’s whitepaper on Goal-Driven Performance Optimization, you will notice that we define performance using the combination of three separate terms. You really want to read the paper, but let me summarize it here:

  1. Response Time – This is the time required to complete a desired task.
  2. Throughput – Throughput is measured in tasks completed per unit of time.
  3. Capacity – The system’s capacity is the point where load cannot be increased without degrading response time below acceptable levels.

Setting and meeting your response time goal should always be your primary focus, but the closer throughput is to capacity the worse response time can be.  It’s a trade-off! Cary Millsap reminds us to think of this just like how traffic slows down with more cars on a highway:

Photo Credit: photoAtlas

Which brings me to my point.

You can actually choose to optimize a system in two different ways – for response, or for throughput. When you optimize for throughput you are relaxing (not eliminating) your response time objectives in order to have more tasks completed per unit of time.

It is much easier to relax response time objectives if the task is not user facing, which is why I often see applications and suggest that they convert a task that happens in the foreground to instead be sent to a message queue, or Gearman.  Or in plain English: The same MySQL servers can achieve  much more work, if you allow the potential for each individual task to take a little bit longer.

Entry posted by Morgan Tocker |

Add to: delicious | digg | reddit | netscape | Google Bookmarks

Powered by WordPress | Theme: Aeros 2.0 by