After the first few month of being out in the open source field I decided to take a look and see what are you – the community – are doing with our software. So I took a pick at sourceforge.net statistics page and noticed that we already passed the 31,000 downloads since April 11^th – our launch date

With a record breaking month in July with 7,526 download in that month alone.

So, then I looked at the last 2 month and filtered the packaged to include the stable version 8.0.5 and then the unstable version 8.1.0. And I found out the an average of 20% of you are downloading the latest version of WebLOAD and (hopefully) using it.

And if I add it to the 3026 registered users on the forum and 1361 posts we get an active community. The best thing about the community nowadays, is that it is almost self managed. We have users helping users and we at RadView mostly listen to you and watching what you are missing and what does not work well for you so we can fix it and make your life better.

So, keep the feedback flowing and try to use the latest version as possible – it mostly contains features you asked for or bugs you reported about.

We are all in the business of searching for our X point or better said, what is the X point relevant to our system?

Let me explain…

Software systems behave similar with response to growing number of users using the system. The following graph represents that behavior

The first few users usually improve the end users experience as more and more data is loaded into the system memory and the system is wormed-up. This is why testers are encouraged to let the system run for the first few users before starting measuring the response time. From that point forward, we will see a very slow increase in user response time as we add more and more users to the system. All this until we reach the X point of our system, after which the response grows exponentially until the system brakes.

So how does this affect the way we do testing?

This graph shows us that as long as we did not reach the X point of our system in the server side, we are dealing with performance tuning of our client side code (see YSlow by Yahoo! And client side performance testing) after we reached the X point we have to identify the bounded resource limiting our application to scale higher if needed.

This leads us to the great question of how to find the X point of the system? Or as we phrased it – the search for your X point.

The search for your X point using goal oriented testing

The first thing about testing in general and load testing specifically is to have goals!

• My system should serve N number of users
• User experience goal: The response time for a user should not pass 5 second for 75% of the requests and 7 seconds for 95%.
• IT goals: CPU should not exceed 95% utilization and average of 75%.

The above are all common example of goals some represent end user experience, others IT oriented goals and obviously you can come with many more. Now once we have these goals, do we plan our test around them or do we take the trial and error approach and see when these goals are breached?

In most cases I heard of, we tend to do the second. We start with 100 users and run our test, then gradually grow the number of users until one of these goals was beached. I think this is the time we change or approach and save us time by moving to goal oriented testing.

In the post “To test or not to test? Client side load testing of Ajax application” I reviewed a long debate we have in the company regarding the need to test client side activity with the ever-growing use of AJAX applications. The bottom line of the post is that we need to separate the server side load testing from client side performance testing. This conclusion got reinforcement after hearing Yahoo!’s presentation about performance and load testing. It appears that Yahoo! has two separate groups, one dealing with server side load testing and the other with client side performance. The last is the one which just published a new tool named YSLOW.

Although separated, both tasks need good tools.

So, you can take a copy of WebLOAD OS and run a single user or use WebLOAD professional with probing client configuration and monitor the following measurements:

Since WebLOAD works on the protocol level, these measurements represent the activity from the moment the request left the client until the response got back from the server. When this is used in combination with Firebug and YSlow you can understand where your client side bottlenecks are and even get recommendation for performance tuning.

The above screenshots shows how Firebug filter only AJAX based activities and show both the network traffic and size as well as profiling of client side activity. Using this information with WebLOAD measurements above you can really understand where you spend time in your code…

With YSlow, you can even get recommendations using Yahoo!’s best practices to where we can better tune our client side activity.

If you have comments or feedback using these tools together, please share them with us in the forum or as comments…

I have been preparing for our coming session at CAST2007 (http://www.associationforsoftwaretesting.org/conference/sponsoredevents.html) and thinking of the different topics we want to discuss with the performance tester community and while doing so I thought why not widen the audience and publish few of the questions in hand here on my blog.

The general idea for such a focus group is to present few of the themes we are debating between and get the team comments and suggestions. Although in the ideal world we wouldn’t have to choose, in the resource bounded reality which we live in we don’t have that luxury.

As Robert Frost wrote in the “The Road Not Taken”, once one takes a certain road, there’s no turning back, although one might change paths later on, they still can’t change the past.

Where should we invest? Question 1:

Should we build a better load generation tool and leave performance analysis to the performance tester himself? Or should we add more tools and support for analysis of the results and identify not just the performance problem but also the source of the problem?

Where should we invest? Question 2:

Who is the performance tester? Can we lower the barrier and make performance testing more accessible?

I recently got a nice observation regarding performance tester profile from a colleague Corey Goldberg

“.. and an observation:

From several years of reading the OpenSTA mailing lists and talking to countless performance testers using a variety of tools, I realize there isn’t much middle ground for performance testers.. they tend to fall into 1 of 2 categories:

1. “I understand HTTP and scalability.. I know exactly what I want a tool to do.. I am a proficient programmer”

2. “I have no idea how to program. I have no idea what performance even is, let alone how to generate load or monitor it… my boss assigned me this task and any tool I choose will be shelfware within a few weeks”"

So going on with these lines here is the question:

Where should we invest? Question 3:

The next question refers to the type of tool we want to see. These days everybody are talking about collaboration and with the Web 2.0 hype and user generated content it is becoming the magic words for hiring people, getting investors funds and publishing PRs. But is this what the performance tester needs? Do we want a collaboration tool or a desktop application?

Where should we invest? Question 4:

I had to go through few years of therapy before I realized that sometime being smart is better than being right J - and this refers to some debates we had around SOA testing in general and Web Services support in specific.

Obviously in ideal world these two should be the same but from previous experience they are not always so. So why smart and not right? Because if the developer is not looking for interoperability and only using WS as a internal client-server remote procedure call, than the load tool should help him generate load using his client stack tool of choice and not bother him with spec compliance issues.

A side note regarding Web services is the place for web services test during the life cycle of the system. We should strive to make sure that the script generated from traversing the WSDL while testing the WS as a standalone component early in the cycle match as much as possible to the same script recorded while testing this web service through end client application as part of the overall system test.

As usual comments are welcome…

Just found out that Elisabeth Hendrickson the blogger behind Ruminations blog has posted a nice blog about WebLOAD move to the open source. You can read it here.

Being one of the executives at RadView I can vouch that the quotes in the blog are not far from the true.

“Executive #1: “Let me get this straight. We sell a product. By selling this product we make money. Remember money? That stuff that pays your salary? And you’re telling me you want us to give away the product for FREE? AND you want to publish the code - our intellectual property - the ’secret sauce’ - ON THE INTERNET?”Executive #2: “Yup.”"

I can also add that being an open source commercial company is not a one step process; we are constantly reviewing our model while listening to the community feedback, all in the effort to provide the best tools, knowledge and support for both our open source community users and commercial customers.

As always, if you think we could do better, do not hesitate to comment here.

Everybody is talking about Ajax, some do it, some use it and some just talk about it… a recent visit to the Ajax world conference, definitely proved that with the amount of activity around Ajax and the different Ajax frameworks a testing solution is also needed.

…So we took our load testing tool, WebLOAD and tried it with different Ajax applications and not surprisingly it just works! – We have to thank our protocol level recording for that. Not only it just worked we also published a WebLOAD HOW TO document helping tester understand load testing of Ajax application and how to use WebLOAD to do that.

…Apparently this is was not enough! When talking to the different Ajax framework vendors they all asked the same question – can you load test the client side activity? This blog is about a bit different question – Should we load test the client side activity?

The first rule with testing is defining the boundaries of our tested system apparently they are not the same when we run functional testing and when running load testing. Functional testing involves testing both the client and the Server together, while load testing involves testing each of these components separately. Adding client side load into the equation will divert the results because the client processing depends on the client machine (CPU & Memory) which is not simulated using virtual client load testing model. Note that I am not saying the client side processing is not important I am just claiming it should be tested separately from server side load.

Another good argument for separating the client side performance testing from the server side load testing is SOA. The basic idea of SOA is to re-use services between more than one applications, this means that the service should be tested based on his contract and regardless of the specific consuming client application.

Automated functional testing process is most often not cost effective to most organizations. Recent surveys have shown that over 50% of enterprises do not invest in functional testing automation tools and new buyers in this area are about 5%. I suspect that this is driven by the fact that on one hand the test code needs to be maintained all the time and on the other hand the same process can be achieved with manual work of relatively cheap men power. Inventing some technology that will measure performance of the client side while automating the action of the users will probably suffer from the same results. It is much cheaper to run a good load test of the server side (using WebLOAD obviously) and run few manual functional + performance testers of the client application.

After a long journey we finally shipped our Open Source project and WebLOAD Open Source edition. One of the key debates we had during this process was how to embrace the open source model on one hand and yet maintain a valid business model for us as a commercial company. This is what this post is about…

So what is the challenge here? We see three different business models in today’s open source projects; the first is what I call the RedHat model where the business model is based on service only. The Open source project (Fedora) is a superset version of the commercial version (RHEL) – all that you pay for is service. This might be good for some companies and some areas in software where certification and service is more critical but other areas might require a different model. The second model is what we call the MySQL model – or dual license model, in this model the product is given to the open source community as a freeware, although you can get the code as well, if you want to do any commercial usage of the product you must buy the subscription from MySQL. The last model is the Add-on model, similar to Zend and PHP relations where the commercial company wrap and add features to the product which provides a more productive and rich environment.

WebLOAD Open Source, the community version, and WebLOAD Professional, the commercial version, are based on the add-on model. The Open Source edition is a fully featured edition which by itself should be superior load testing product. We are looking for vest distribution and adoption of the open source edition. We at Radview are committed to continue develop WebLOAD Open Source together with the community and drive new features into it. The guiding rule for separating the community version from the commercial version is to ensure that:

1. The professional version supports additional commercial environment such as load testing for Oracle Forms or Adobe Flex DS
2. The professional version provides features which make your testing experience more productive saving you time and money.

The product comparison table can be found on http://www.radview.com/product/Editions-Comparison.aspx

One of the most FAQ is the distributed load question, or in other form, does the open source version limited in the amount of load it can generate against the SUT?

The clear answer is NO. There is not technical limitation, beside the size of your load generator (LG) machine, the way you wrote your agenda and the response of your SUT. Nothing in the code prevents one LG to simulate thousand VC. If your LG server is overloaded, nothing prevents you from taking another copy of WebLOAD Open Source and using it side by side on a different load generator against the same SUT. So what does the distributed load feature of the professional version is all about? It is about productivity, it enables you to use a single console and define the different load generators machines, set their different load configurations and collect the results back to one monitoring console. None of the above is something you can’t do with the open source version it just takes you longer.

Another good example of the difference between the open source version and the commercial version is our coming WebLOAD Reporter. The new version of the reporter is based on Eclipse RCP framework integrated with Jasper Report and provides a new productive reporting environment. This new version is going to be released to the open source community, replacing the existing Reporter which is not yet open source. We did not have the time and resources to migrate all the code to the open source community, the new version of the tool is first released to the community and only then integrated into the professional version. The first version of the reporter in the professional version will also have a larger set of pre-defined reports in the report gallery.

These are exciting days at RadView. The company is launching its flagship product, WebLOAD as an Open Source offering and building itself as a commercial open source software company. In my career I’ve witnessed several incubations/inceptions and as always these are exciting moments.

I reflect on the first product launch at Mercury Interactive. The founding group had in mind a solution for software testing that resembled CAD workstations – an industry we all came from. As a result the product was built as a software/hardware combination around a PC server running OS2 (remember?) – quite a cumbersome way to tackle software testing. Mercury has corrected that issue since but what we got absolutely right was the need to install test automation tools in order to tackle the no. 1 issue of software – its quality! Mercury grew as a company since and was just recently sold for $4.5B in cash to HP, a price tag nobody even dared to dream of in those early days of 1990. Even more importantly is how the software quality industry has matured and grown – from a bunch of nutty professors rambling about code coverage to a professional industry that demonstrates its strength almost daily in publications and conferences around the world. The Software QA as a profession and as an important organization has grown up tremendously and I take personal pride in my own small contribution to that trend.

With all that progress the issue of software quality still looms over the software industry. As we are striving to build more complex systems where software plays the key technology role we are facing the issues of quality. Poor quality daily rears its ugly head – by miscalculating the fare for the subway as happened in Hong Kong when I last visited that lovely city (and otherwise one of the best underground systems I’ve ever used) or in the collapse of a web portal for one of the major mobile operators on the day of the launch of a major ad campaign as it just happened in my home country Israel.

So we still got a long way to go and in my view good tools are a key. You can never deemphasize the importance of good testing tools which are widely deployed and are at the hands of every tester and developer involved in building the system. The reason is the simple truth that you have a much higher chance to get a high quality and functional system if you test early, often, at the smallest available component and you make testing a regular procedure invoked at every build of your system. This approach calls for widely available tools that will be tightly integrated into the development process and the best way to do that is to open source the tools.

Today RadView is making a major step in support of open source testing by offering its flagship product, WebLOAD as open source. WebLOAD is well tested in the market with more than 1600 deployments since its inception in 1998 and has been used to test major web applications like the 2002 Salt Lake City Olympics, among others. By offering WebLOAD as open source RadView hopes to foster innovation around the product line gaining product ideas as well as implementations from the large community of developers and testers that can now experience the product first hand. RadView is in the process of developing the next generation IDE and UI for WebLOAD based on the Eclipse platform which will make the product even more readily available to the development community. RadView also hopes to integrate this important tool with major open source distributions like Eclipse and JBoss as every such deployment requires a performance testing solution to verify its correctness.

Most importantly RadView would like to generate a movement that will make available first-grade and well integrated open source testing tools in order to form a complete testing solution available for every developer and tester around the world. RadView hopes to create test standards agreed by all, to create the notion of a “testable application” (more about it in future blogs) assuring the quality of every software system out there.

This is indeed an exciting journey, hey – it feels like 1999 all over again! I hope you join me on this ride and I would love to hear your comments.

Ilan

RIA is not just about new user interface and sync\asynchronous web services calls, advance use of RIA will include different type of push models also referred as messaging\pub-sub\real-time type of scenarios. When it comes to performance testing we must make sure we define the SUT boundaries before we define the solution. The boundaries has not changed it is still the whole server as a black box. This means that when come for load testing, we have to disregard any activities happening inside the server and simulate\test activities coming into and out from the server.

Let’s take a sample NBC game notification scenario were few clients registering for event on a specific game and waiting idle for data to arrive. Once the game started an outside entity send an update request to the server which later triggers an event to be sent to all users.

The following diagram illustrate the sequence of events and calls between the system and the clients.

Note that I am still referring to the SUT as a single unit assuming all requests are coming from an outside source.

How would we load test such application?

We need to run two agendas for this system, one representing the clients registration and waiting for the notifications and the other representing the external server sending the game results.

These two agenda need to be synced, so for example the game server will not publish the game results before all clients have registered. Last, the clients agenda need to wait for the events to arrive. For this to happen we need a protocol which support such scenario, HTTP by itself does not support it so it can’t be done with the standard HTTP support, but in the future we assume such application will evolve such as Adobe flesh protocol and with them the ability to wait for evens inside WebLOAD.

In the time been, you can use your own client proxy implementation written in Java or COM and call it within your agenda.

Development framework has been around for years, they are all about standardization and acceleration of the development process, but what about a testing framework? Why would an enterprise want to implement one?

The answer to these question relates to test automation question, many organization have question in the past the need and cost effectiveness of test automation, in functional testing one can always replace test automation with manual testing, load testing is always done with automated tools or not done at all. The question of how to make load testing cost effective is still valid and load testing framework is key part of the answer. If we look closely on the load testing architecture we can divide the system to three parts; the client GUI application, usually implemented as browser application using static HTML or rich internet application such as Ajax applications. The second part is the internet protocol available to access the system under test, this is usually HTTP/HTTPS but also include multimedia protocols such as RTP/RTSP or “Web 2.0” protocols such as RSS feeds and last is the system under test (SUT) which is the subject of our load testing. This is an important distinction we have to make between function testing where both the client and the Server are being tested together and load testing where each of these components are tested separately, adding client side load into the equation will divert the results because the client processing depends on the client machine (CPU & Memory) which is not simulated using virtual client load testing model.

With the growing adoption of service oriented architecture (SOA) we have a new layer that worth mentioning here in this context – the system API. In the SOA world the system API are the set of services offered by different sub systems and consumed by the client application usually in the form of web services. System API are not limited to SOA web services implementation, any reasonably stable system has some form of separation between the GUI and the business logic, these business logic functions can be access via stable System API. Since these API’s are considered more stable from the trendy GUI application consuming them, a method of load testing is required which is not affected by the ever changing GUI application. This is where LOAD testing framework comes into play; the load testing framework allows you to build new load test agendas without recording new script, thus eliminating the effect of changes in the UI, and only using the pre-defined systems API calls like any other callable object within your JavaScript code. Inside the agenda you access external resource to get your parameters for each of the calls.

Any professional QA team should start thinking how can they build such framework and reduce their TCO and provide better and faster service in their organization. Radview with the new release of WebLOAD 8.0 provide the best answer for such effort. Stay tuned for a comprehensive white-paper coming from RadView R&D team.