Photo contact sheet from command line

Recently I needed to inventory a bunch of equipment at a client so I went armed with camera and took a bunch of pictures and notes and came home. Ultimately what I wanted was one or more files containing thumbnails of each image along with the filename below it to include in the report. I suspected that there was a command line way to do this – in Windows – and there is. Through the power of Cygwin (a surprisingly complete Linux CLI environment that runs on Windows), the magic of Imagemagick and the guidance from the post below, I created my own recipe.

http://blog.patdavid.net/2013/04/using-imagemagick-to-create-contact.html

Here is my command line to transform a directory of .jpg files into a few (3 in my case) contact sheets also in jpg format.

montage -verbose -label '%f' -font Helvetica -pointsize 16 -background '#ffffff' -fill 'black' -tile 3x4 -define jpeg:size=400x400 -geometry 400x400 -auto-orient *.jpg contact-light.jpg

Read the blog above and Imagemagick docs to explain the switches. I did change a few things in the blog’s example. I bumped up the resolutions of the images, (and hence the contact sheet) a bit, made it black text on white, bigger font, and limited the images to 3×4 pics per page using the tile options.

Since I used Cygwin on Windows I installed all the Imagemagick utils through the easy to use Cygwin installer, as well as Ghostscript and all the fonts I could find for it.

Below is the result.

contact-dark2-2b

Advertisements

ArcMap 10.2 slowness

We’ve been experiencing some a lot of slowness with ArcMap 10.2 across many  Windows 7 (64-bit of course) Pro and Enterprise machines. So my colleague has done a lot of research, phone calling (with Esri), and troubleshooting to arrive at the following helpful information. If you having some weird and annoying slowness in 10.2 (or perhaps even 10.1) some of the options below may help you. We’re not convinced that these tips have solved our problems, but they seem to help.

1. Try not to use Personal Geodatabases (PGDB) (.mdb) format.

We wish that Esri would issue proper warnings about using PGDBs rather than just exhibiting odd behaviors and slowness. But, display, selection,  and geoprocessing issues exist when using .mdb files in 10.2 (issues were also encountered in v10.1, too). These issues seem to be related to performance (e.g., on machines with both spinning disks and SSDs: about 60 seconds to open a 100 record, 500kb shapefile, another 60 seconds to export (locally)) which then may or may not contribute to the overall flaky behavior that we get in the desktop (i.e., random features just not appearing in the map).

Apparently switching between geoprocessing in the foreground (32-bit) versus the background (64-bit) switches between 32 and 64-bit processing. The default is background processing.This makes sense now finding out that personal geodatabases files are unsupported in v10.2 with 64-bit AND background anything. See the docs below. The upside is converting .mdb to file geodatabase (.gdb) appears to have fixed the problems.

ArcGIS64-bit_docsSolution: Don’t use PGDBs and if you do, set everything to run in the FOREGROUND, which forces operations to be done in 32-bit mode. Switching to foreground brought the speeds back to near instantaneous (un-checking the enable background processing in the screen shot below).

ArcGIS64-bit_docs2

Given that SQLite (Spatialite and Geopackage) are now or soon will be nearly first-class citizens for storing for use in Arc*, if you need true database functions (like joins, views/queries) the various SQLite formats might be a good eventual replacement for PGDBs.. and dare I say even Shapefiles! Much will depend on how seamless Esri makes using them. Right now it’s actually quite SEAMFUL.

2. Don’t update metadata if you don’t have to

Also the Arc Map default is to update metadata anytime you touch a file. We turned this off and likely has aided in performance enhancement as well. The improvement seemed to be “noticeable” but not what you’d call terrific.

3. Rename your home

This is a more hard-core version of replacing normal.mxt. But it seems to be what is needed if that doesn’t resolve additional flaky/slow behaviors.

Be aware that renaming the Esri folders is a factory reset of ArcGIS, and therefore all third party tools, custom scripts, and/ or custom toolbars currently installed will need to be re-installed as a result. Also, you will need to re-connect all existing folder connections.

***Renaming the esri Folder in the C: drive***

1. Open Control Panel > Appearance and Personalization > Folder Options > View Tab

2. Select the radio button to Show hidden files, folders, and drives

3. Right-click on the Windows Start Icon > Windows Explorer

4. Navigate to C:/Users/YourUsername/AppData/Roaming/esri

**If the AppData folder is not visible, you may need to perform the following steps to display the general toolbar: Organize menu > Layout > Menu Bar. Then to display the AppData folder, choose the Tools menu > View Tab > Show Hidden files, folders, or drives > Apply > OK

5. Right click on the esri folder and select rename. rename the folder.

HTH

Postgres database from itis.gov

This might save you some head aches. The postgresql database dump of the taxonomic universe (according to the USGS) from http://www.itis.gov/ is provided in a latin1 character encoding (specifically it tries to create a database as ENCODING = ‘LATIN1’ LC_COLLATE = ‘en_US.ISO8859-1’ LC_CTYPE = ‘en_US.ISO8859-1’ ). In order to bring this dump into a modern Postgres instance you’ll likely have to deal with the encoding being different in the dump versus your system. Rather than changing the locale of your entire database (which is likely UTF-8). It’s easier to change the encoding of the SQL file they give you.

Using this excellent reference:
http://stackoverflow.com/questions/64860/best-way-to-convert-text-files-between-character-sets

I went right for iconv and this worked for me on my ubuntu 12.04 (after installing recode which might have brought in some additional encodings that weren’t there before).

iconv -f latin1 -t UTF-8 ITIS.sql > ITIS-utf8.sql

Then you have to edit the SQL dump file to remove references that enforce the encoding so that the database will just use its defaults when you load it.

PostgreSQL/PostGIS and ArcGIS ArcMap 10.2

Of course good things come to those who wait (at least that’s the wisdom that I pass along to my kids). We expect that after we’re done waiting we will get something awesome and worth the wait, right? But sometimes what we end up getting is a little underwhelming and as adults we’re forced to just accept it with a shrug and move on. I feel like that right now and it will become clear why by the end of the post.

But, for now I rejoice in the good part which is that I can finally connect to my handy Postgres/PostGIS database and see my data there using ArcMap. I can query, style and do most other basic things that I have tried. To see how to do this you may want to take a look at the docs. In this case I installed Postgres 9.2.4 and PostGIS 2.0.something using the EnterpriseDB installer and Stackbuilder (http://postgis.net/windows_downloads) . Esri will give you Postgres 9.2.2 which still has that nasty security flaw in it that everyone else upgraded from months ago. 9.2.4 seems to be working fine though using the 9.2.2 client drivers.

Note that Esri keeps the Postgres client drivers away from you unless you are “authorized” through your customer number to have these free/Free drivers that are not published by Esri. You must download them separately from the ArcMap install. Here’s mine PostgreSQLClientLibs922 if you can’t get access to them cuz you are trying the demo of ArcMap 10.2 – which makes you not worthy of gaining access to these drivers.

postgres_drivers

BTW, if you are a better reader than I you would immediately note that the bit-ness of the drivers should match the *client* software that you are installing them into, NOT the database you are connecting to. So, by installing the 64-bit database server, then trying to get ArcMap to use the 64-bit drivers I was doomed to suffer through the useless Esri errors for an hour until I had a face-palm moment. You must give ArcMap the *32-bit* drivers.

Bug alert: I can’t reproduce this right now, BUT when I first created the connection ArcMap complained because its default database port for Postgres came up as 54321. If you get an error while first connecting, recall that by default Postgres’s port is 5432. So you need to force that in the connection “Instance” field by entering the machine followed by a comma, then the port number like 127.0.0.1,5432 as shown below.

db_port

Once you connect this little issue goes away <shrug>.

So I finally got the DB connection working and was very excited. Here’s a pic.

database_data

You see some poly data from PostGIS (make sure that you set the SRID when loading) and a few points from my handy Spatialite database (no database connection needed to display these Spatialite data. Must be the GDAL/OGR baked right in). Bill Dollins explains a little more about using Spatialite data in ArcMap in this post.From the docs it seems that Esri is supporting Spatialite 4.0, which may also support the current 4.1 as well.

Here’s the similar view from QGIS.

qgis_post

 

Ok, so now I’m feeling pretty good about myself. How about trying to edit some of these non-geodatabase data in ArcMap? Without too much drama, the short answer seems to be that you can’t. When Esri says “you may USE the database data in ArcMap” they mean you may VIEW it – and only view it. You may not edit PostGIS data directly in ArcMap.

no_editing

 

It’s not the end of the world since I can do most things in QGIS running side-by-side. But it’s annoying.

I’m pretty sure that the Spatialite layer can’t be edited either. Despite setting all layers displayed to EPSG SRID = 4326 (WGS84) ArcMap still feels that they layers are not in the same coordinate ref. Then goes on to complain about not being able to edit the layer.

postgis_coordrefspatiliate_crefstillnoedit

So, I’ve been waiting a long time to be able to use my non-ESRI geodatabase database geodata in ArcMap with all the functions I expect including editing. But I guess I have to wait a bit longer… sigh..

Here is a picture of the geodata as seen by the free and included Postgres IDE pgAdmin III

gis_data_indb

UPDATE: I should mention one other thing. I did make some edits to a layer using QGIS and was hoping to see them appear in ArcMap after committing them to the DB. Alas, this did not happen and I had to remove/re-add the layer for the edits to appear. So, it seems that perhaps ArcMap is caching the layer (perhaps in a little hidden FGDB) when it loads.

The recommended approach for editing PostGIS data through off-the-shelf Esriware seems to be through the REST services and a feature service provided by ArcGIS Server 10.2 (since the Spatial Data Server (SDS) ) is now deprecated and appears to be rolled into Arc Server itself. I do hope that these edits made through Arc Server are immediately available through the online stuff and will not require refreshing any cache.

 

Resources:

 

 

I Love LIDAR

Over the years I’ve tasted good bear, coffee, and hot cocoa. Now I have a hard time drinking the cheap stuff– though I try not to drink expensive wine for fear of losing my wallet. I’m about to get another expensive taste: LIDAR data. I’ve never worked with it before, but it’s incredible!

I'm never digitizing without it again!
I’m never digitizing without it again!

Spatialite wishlist for 2013

The Spatialite project and its family of products is progressing and gaining a larger following by the day. Growth seems to be coming from its apparent target audience – mobile developers (though I rely on it for desktop use). This is likely to only snowball as Spatialite acceptance increases and it gets woven into increasing numbers of projects and  workflows. Sandro and Brad and maybe a small handful of others are making methodical and incremental progress advancing it. It’s quality is improving and its features and capabilities are growing like the weeds in my yard. But, Spatialite and its team can be so much more. So here is my wish list based on more than two years of watching and using it.

First let me say that I’m truly writing in the spirit of constructive observations. I don’t want to be critical of the current project members as their patience and generosity are amazing. But beyond that, Sandro is the father of the project and,of course Spatialite is his to run any way he wishes.

So here are some wishlist items.

Simpler, default use of spatial indexes in queries: Right now in Spatialite you have to use sub-queries to use spatial indexes in your queries. It’s not the end of the world, and it does allow you a certain flexibility in crafting your code. But, they are easy to screw up and frankly it’s more typing for something that you want to do often nearly always. Other spatial DBs just use the indexes by default.

A GIS GUI that makes full use of the capabilities – The core Spatialite offers lots of cool little utility functions (i.e., SanitizeGeometry, ST_IsValid, etc. in addition to the standard geoprocessing ones and they accessible from the command line via SQL statements. Now I’m fine with the command line, but I haven’t done GIS with flashing cursor in years.  Sandro provides a nice little set of GUI tools, and I think he likes writing them. But they don’t expose ALL of the latest functions as I’m sure these take time to include. I’d rather the vast array of libspatialite functions get exposed through a single, go-to application instead of having to keep a few GUI utilities laying around or bounce between flashing cursors and mice. So, I’m really hoping for GUI controls in Qgis that will let me harness all of the power in Spatialite functions from within Qgis without having to keep both Qgis and a terminal open. This is going to get even more critical if the Geopackage spec gets cleaned up and goes anywhere.

Larger Developer Community: Spatialite really seems to be a labor of love to Sandro – and it shows with his support of user questions on the support list on Google Groups. Similarly, development seems to be guided mostly by Sandro’s needs — which is fine to an extent. But the development road map lacks transparency – which is to say that it’s not missing, just needs some more daylight and/or planning. Users might have some inkling of major new initiatives that Sandro and Brad are working on when they have the time to write about them in an email or a wiki entry. But there is no complete public road map, no true work log, and mostly we hear about new tweaks such as SQL functions only after they are released. You can kinda see what HAS happened if you watch the Fossil timelines – But this isn’t ideal. So, I am hopeful that this project will mature into one having a larger, interactive, transparent, diverse and functional developer community and that Sandro can and will allow other minds to influence the direction that future Spatialite development takes. In short, I think the guys need help with the standard mechanics of OSS projects (development, documentation, outreach, etc.).

Documentation: There used to be extensive documentation in verbose narrative form on scattered, stand-alone pages on the Spatialite site, but much of it is outdated. Sandro tries to keep up documenting the new features and changes, but I do think that more documentation would help. It would also be help if the documentation were provided in a more structured (perhaps even hierarchical) format instead of scattered across the Google Group email list and un-structured partially linked, non-hierarchical pages in the old Spatialite site and the new Fossil-based wiki. I’ve always been a big fan of the documentation provided by the folks at PHP and MySQL. I tried once to contribute to writing documentation, but I’m not doing my personal and professional life well enough, and so I didn’t get very far.

Better support for MS VC++ in general and non-manual support for system.data.sqlite: Sandro and the other guys don’t like doing Windows. I get it. But people really want to be able get Spatialite happily integrated into their .Net development life – and these are people who don’t do ./configure make make install in their sleep. Sandro has posted notes for working with his project files in Visual Studio, but they frighten Microsoft-only folks. Sebastian* found agony with this and created some additional work (https://bitbucket.org/mayastudios) to help him do Windows. But I hope that someone steps up to help the current project members maintain parallel builds and projects targeted at better supporting Windows and other platforms. Sandro and Brad maintain the core libs and do what they can to assist integration with other platforms. So some dedicated downstream maintainers for other platforms and builds are really needed.

Less is more: Honestly, Spatialite already has just about every spatial functions a person could want for 80% of their needs. It’s really quite complete that way. So, my personal wish is that the supporting documentation, utilities, interfaces, linkages and platforms get a little more attention now. But I think now that we are the reference implementation for vectors in the new OGC Geopackage spec more energy will likely be diverted there instead of shoring up these existing ancillary things – at least until new developers and helpers get involved.

So these are just a few items on my wishlist. We’ll see if the future takes us to places where they addressed or become irrelevant.

————————————-

* From Sebastian about his bitbucket “It contains a VS 2010 solution to compile Spatialite. You need Mercurial though to check it out. Just downloading it from bitbucket won’t work because the repository uses sub repositories (which aren’t included in the download).” (you need to install mercurial to successfully work with his repos. If you’re just looking to clone (check out) the repository, use “hg clone [url] [dir]” or use TortoiseHg): The new tip (newest revision) builds Spatialite 4.0.0 on Windows.

  • The repository spatialite-lib-windows just compiles the native (unmanaged) “spatialite.dll” – along with “sqlite.dll”, if you need it. The same is true for the “spatialite-lib-android” (former “spatialite-android-lib”) repository, just for Android. The repository “spatialite-android-java” (former “spatialite-android”) is obsolete.
  • SQLite.Net started as a port of the Java SQLite bindings with the goal to allow it to be used cross-platform. At the time of writing, I was fairely new to SQLite and I couldn’t get “system.data.sqlite” compiled (AFAIK). So I sticked with my own implementation.
  • >Does sqlite.net let me call spatialite more directly through .Net?
           – No, that’s what spatialite.net is for. It allows you to directly bind/obtain geometry objects (provided by NTS) to/from the database.
    >Are you also the author of the other projects I see at Maya Studios?
           -Yes, I’m the author of all of them.
  •  If you’re just looking for the Spatialite binaries, you can download them here: https://bitbucket.org/mayastudios/files/src
  • Sebastian just recently posted this to the Spatialite Google Group for those needing Spatialite on Windows.
  • Also see: [SpatiaLite-Users] SpatiaLite v4 with c# on VS2012. 

    Vittorio Maniezzo via googlegroups.com. 

    Apr 10, 2013

  • To spatialite-users.
    As a couple of people asked me for this, maybe there are more out there interested.
    I uploaded on my server (at http://astarte.csr.unibo.it/snippets/testSpatialite4.zip )
    a very bare VS2012 c# project using SL 4.0.0. It’s not that much complete: the 32 bit version works fine, the 64 bit not yet. But in the near future I won’t have time to work on this, so maybe someone can help.
    Cheers,
    Vittorio