Vancouver+ for FSX status and test plan overview
Holger, Scot and I are currently testing/refining the configurator/installer. Once we are satisfied with it, we will release the first beta to the test team in the next few days. Testers have been chosen, the testing forum is set up, test plan written, and test allocations done. Testers have signed up to check the product by geographic sector, and by functional area. While the team is testing, I’ll finish off the CYNJ buildings and some other loose ends.
Our approach is to provide the beta team with a fairly well tested release. We’ll fine out how it works on different platforms, including some older machines that we’ve included in the test group. We will minimize the number of beta drops (test releases) by running a set of regression tests prior to each drop.
For those interested in a deeper view into our testing procedure, here’s our simple test plan. Note that is directed to the beta team only, and is provided here should it be useful for other developers.
Vancouver + FSX – Test Plan and Sector Map
Because of the large scope of this project, both in terms of geographic area and features, we will be asking folks to sign up to do specific types of testing. This way we will help ensure nothing falls through the cracks.
There will be 7 geographic test regions, Alpha through Golf, and several non-geographic functional tests.
We also will be doing a simple form of regression testing: when we specify a new beta release is an RT beta, all tests must be run again. This minimizes the chances that fixes and enhancements do not cause collateral damage. We will of course work to minimize the number of complete test runs we do. For those who are interested, I have a document available entitled, “Regression Testing for Flight Simulator Addons”.
Of course, we still encourage mucking about, flying at will, and just having fun. That may find problems we haven’t thought of.
Here are the geographic test sectors (see below):
Some notes on reporting:
• please use screenshots with coordinates enabled (ctl-z), limited to 125kb, 6 per post (though multiple posts are possible), and (please!) max. width of 800 pixels. If you don’t want to hand-edit your screenshot use something like Grab-Clip-Save http://www.simviation.com/fsutilities_gen3.htm
• please report in the appropriate sectional threads (see below) and avoid starting new threads unless it’s a very general (or very specific) issue
• report time of day/season
• for AI traffic monitoring, for those of who have FSX Deluxe, use the Traffic Toolbox
• please ensure you have looked over existing bug reports carefully
• maps for British Columbia: http://toporama.cits.rncan.gc.ca/toporama_en.html or here http://webmap.em.gov.bc.ca/mapplace/minpot/general.cfm (the second link requires a free plug-in for the interactive map). http://maps.gov.bc.ca/ and click on “Land and Resource Data Warehouse Catalog”
• googleEarth http://earth.google.com/
• VanMap, for detailed downtown airphotos: http://www.city.vancouver.bc.ca/vanmap/o/orthophotos.htm
• Burnaby, for detailed Burnaby airphotos: http://webmap.city.burnaby.bc.ca/publicmap/viewer.htm
• Richmond GIS for detailed Richmond airphotos: http://map.city.richmond.bc.ca/website/gis/viewer.htm
• Photos: http://www.webshots.com/search?new=1&source=mdocsheader&words=
• Other photos: search at http://www.google.com/imghp
• Oblique air photos: http://www.globalairphotos.com/
• Buildings: http://www.skyscraperpage.com/ and http://www.emporis.com/en/
Known Issues – please don’t report unless noted below:
• List coming
Test Section Overview
Notes on Test sections:
• Please report issues in the thread provided
• Delta and Echo areas have many features, it is suggested that if you sign up for one of these, you don’t sign up for other areas
• you can report issues outside your chosen area, but please use the correct thread
Sign up by section:
• Section 1a – Geographic Area Alpha
• Section 1b – Geographic Area Bravo
• Section 1c – Geographic Area Charlie
• Section 1d – Geographic Area Delta
• Section 1e – Geographic Area Echo
• Section 1f – Geographic Area Foxtrot
• Section 1g – Geographic Area Golf
• Section 2 – AI
• Section 3a – Airports – land-based
• Section 3b – Airports – Seaplane bases
• Section 3c – Airports – Heliports
• Section 4 – Third party compatibility
Everybody will do:
• Section 5 – Performance
• Section 6 – Installer/Configurator
Test Section Descriptions and What to Look for
Please report any crashes and shutdowns with as much detail as possible so that others can try to recreate the situation.
Please check for and report major frame rate drops and increased loading times.
Suspension of disbelief: Holger’s own measure of success is to fly around and not come across places where the environment looks or feels “wrong”. For example, are specific textures (rocks, forests, water) strange or of the wrong color? Do the shorelines look off? Are the settlements placed well? Anything that makes you go “hmmm…” or “ouch!”
Section 1 – Geographic Tests – by regions (Alpha – Golf)
Sign up by geographic area – See test regions above.
Objects (Buildings, dams, marinas, bridges, other structures) – look for flickering, incomplete or inaccurate textures, blend with context, misplacement
Photoscenery – look for blend with context, edges
Lake elevations and river placement and elevations – look for plateaus, pits, breaks
mistakes in vector line designations – look for roads as streams and vice versa
Roads and railroads – look for misplacement, breaks, river/lake crossings
Shorelines – look for missing, incursions: river meets or leaves lake
Utility Cuts – look for how lovely they look
Landclass – look for mistakes in landclass (eg desert instead of forest)
Autogen building encroachment – look for buildings on steep mountainsides or other inappropriate places
– for your chosen region(s) fly a pattern and check the above in detail
– fly a coarse pattern over the region in each season (Winter, Spring, Summer, Fall) and each time of day (Dawn, Day, Dusk, Night)
– report by lat/long with screenshot with coordinate display turned on (ctrl-z)
– you may be asked more than once to repeat the full test before final release
Section 2 – AI
Seaplanes – look for smooth departure from dock or ramp, and smooth docking or parking. Smooth traffic flow at runways and taxiways.
Small boats – look for collisions in paths, crossing land, etc.
Cruise ships – look for collisions in paths, crossing land, etc.
Helicopters – look for collisions with buildings, land
Testing: use Traffic Explorer to monitor takeoffs, landings and progress.
Section 3 – Airports
Testing: Check ATC, runway designations, parking,
Section 4 – Third party compatibility
Test with any third party addons you have and report anomalies
Section 5 – Performance
We will provide some test flight scenarios and required settings to create some benchmarks.
Section 6 – Installer/Configurator
Install and uninstall, reinstall, repair.
Appendix – Test Regions (see map above)
Test Region (TR) Boundary
Alpha (Howe Sound) 50° 37.50’N 123° 45’W x 49° 30’N 122° 58’W
Bravo (Garibaldi) 50° 37.50’N 122° 58’W x 49° 30’N 121° 55’W
Charlie (Fraser Canyon) 50° 37.50’N 121° 55’W x 49° 30’N 120° 56.25’W
Delta (Vancouver) 49° 30’N 123° 45’W x 49° 11’N 122 °58’W
Echo (Golden Ears) 49° 30’N 122° 58’W x 49° 11’N 121 °55’W
Foxtrot (Hope) 49° 30’N 121° 55’W x 48° 52.03’N 120 °56.25’W
(includes part of US, test that too)
Golf (Fraser Valley) 49° 11’N 123° 14’W x 48° 52.03’N 121 °55’W
(includes part of US, test that too)