Download all of my KMLs

[minstarmin]

Hello,
I am performing a study of wireless security. For this purpose, I really don't want to go through the uploads page, clicking each and every one of these KMLs manually. Looking at the API, I see there's a directory of KMLs that I can download, under /api/v2/file/kml/. Ideally, I'd like to create a Python script that:
1. Creates a list of all my uploaded KMLs
2. Parses their names into an array of strings
3. Uses wget on each one of them
4. Filters through each KML, removes any duplicates
5. Combines the files based on city, state, country, and continent using regex's for GPS coordinates
I have a general idea of how I'll go about doing this. My question is, provided that I have discovered >105k networks with GPS, would I be allowed to perform such an action? I assume that the combined mass of all my KMLs would be relatively large, and downloading them all in rapid succession may be rather taxing on the WiGLE servers. I realize that these files are just sorted text, but still, I think that it would be rather rude of me to download this many files without permission. Please let me know.

Thank you

[bobzilla]

As long as you do it serially I think that'd be ok. You have ~2k uploads so it will take a while, as you suggest.

[fbar]

somewhat related question:
As part of a project to map publicly accessible Wi-Fi on LA's Skid Row, we're planning to wardrive/bike/walk the area (and will of course upload the results). How would we go about downloading KML's of our uploads that highlight the hotspots that are possibly public? I see the wigle map has "Possible FreeNet" and "Possible Commercial Net" filters, but that doesn't seem to be part of the downloaded KML. What's the best way to do this?
Thanks in advance!
(I'm new at this and would welcome tech help if anyone is interested)

[arkasha]

Hi fbar,

Thanks for your interest!

The possible free/pay nets are heuristic anyway - we just use a curated list of known free/pay SSIDs to guess at the status of the network. We'd be happy to share the current list with you (it used to be curated through the Wiki, but we got tired of having to clean the spam out of it) - just email us at WiGLE-admin[at]wigle.net. We have an open-source KML-combiner python script named "kml_filter.py" that buckets points by encryption status on the Tools page - I'd start with that, and modify it to filter by the list of likely free/public nets you'd like to see.

Since we removed those lists when we got rid of the wiki, we should find a new way to make them available - we'll look into a simple API endpoint for that.

If you modify the script to make it more versatile than just an encryption sorting tool, it would be cool to publish it back to the community; let us know if you feel like sharing!

Cheers,

-Ark and the WiGLE team

[minstarmin]

somewhat related question:
As part of a project to map publicly accessible Wi-Fi on LA's Skid Row, we're planning to wardrive/bike/walk the area (and will of course upload the results). How would we go about downloading KML's of our uploads that highlight the hotspots that are possibly public? I see the wigle map has "Possible FreeNet" and "Possible Commercial Net" filters, but that doesn't seem to be part of the downloaded KML. What's the best way to do this?
Thanks in advance!
(I'm new at this and would welcome tech help if anyone is interested)

As previously mentioned by Arkasha, the kml_filter.py script located at the bottom of the Tools page is quite underrated. I've used it several times a while back to check out what encryption was popular in my area - similar to the study I'm performing now. I would love to help you with your project. Please hit me up if you're interested!

[fbar]

Thanks arkasha and minstarmin! Give me a little while to get started and I'll get back to you.

[arkasha]

The freenets list is more gimpy than I'd realized. We should find a new way to crowdsource this, since it's kinda the way WiGLE works!