Arcpy on Josh Werts

Group by query in a file geodatabase

Tue, 16 Jan 2018 13:51:36 -0500

File geodatabase feature classes and tables lack some of the more advanced ability to query that a true relational database supports. Sometimes these queries can be simulated with arcpy cursors; one good example is a SQL group by query.

Typical SQL:

SELECT field, count(*) from table group by field order by count(*) desc;

In arcpy:

from collections import Counter

def group_by_count(table_or_fc, fields):
    """ Returns dictionary containing count of unique items """
    counter = Counter()
    with arcpy.da.SearchCursor(table_or_fc, fields) as curs:
        for row in curs:
            # no need to store as a tuple if only 1 field, just store the value
            if len(row) == 1:
                row = row[0]
            counter[row] += 1
    return counter


def group_by_count_formatted(table_or_fc, fields):
    """ prints out counts of unique values """
    counter = group_by_count(table_or_fc, fields)
    # sort yields highest count records first (order by count(*) desc)
    for key, count in sorted(counter.items(), reverse=True, key=lambda item: item[1]):
        print("{}: {:,}".format(str(key), count))

Example usage in ArcMap Python console (single field):

>>> group_by_count('junctions', 'ImpedanceType')
Counter({u'SmallStreet': 455145, u'LargeStreet': 28714, u'Stream': 9375, u'RailRoad': 1742})

>>> group_by_count_formatted('junctions', 'ImpedanceType')
SmallStreet: 455,145
LargeStreet: 28,714
Stream: 9,375
RailRoad: 1,742

Example usage in ArcMap Python console (multiple fields):

>>> group_by_count('junctions', ['ImpedanceType', 'InfrastructureType'])
Counter({('SmallStreet', 'New'): 318834, ('LargeStreet', 'New'): 28710, ('Stream', 'New'): 18379, ('Stream', 'Aerial'): 5806, ('RailRoad', 'New'): 4043, ('Stream', 'Underground'): 3227, ('RailRoad', 'Aerial'): 1035, ('RailRoad', 'Underground'): 668})

>>> group_by_count_formatted('junctions', ['ImpedanceType', 'InfrastructureType'])
('SmallStreet', 'New'): 318,834
('LargeStreet', 'New'): 28,710
('Stream', 'New'): 18,379
('Stream', 'Aerial'): 5,806
('RailRoad', 'New'): 4,043
('Stream', 'Underground'): 3,227
('RailRoad', 'Aerial'): 1,035
('RailRoad', 'Underground'): 668

These functions aren’t going to be as efficient as a SQL query, but they can be quite useful sometimes for ad-hoc data exploration - especially in the Arcmap/Pro console.

FeatureClass from JSON

Fri, 14 Oct 2016 10:07:44 -0400

I keep forgetting how to do this after several months pass by and it’s certainly not obvious…

ESRI provides a tool for converting JSON to FeatureClass but it requires reading the JSON from a .json file: JSON to Features. That extra step of writing out and reading from a file just seems completely unnecessary.

Here’s the alternative lesser known and not obvious way to do it:

valid_featureset_json = \
"""{
    "displayFieldName": "",
    "fieldAliases": {
        "OBJECTID": "OBJECTID",
        "Description": "Description"
    },
    "geometryType": "esriGeometryPoint",
    "spatialReference": {
        "wkid": 4326,
        "latestWkid": 4326
    },
    "fields": [
        {
            "name": "OBJECTID",
            "type": "esriFieldTypeOID",
            "alias": "OBJECTID"
        },
        {
            "name": "Description",
            "type": "esriFieldTypeString",
            "alias": "Description",
            "length": 50
        }
    ],
    "features": [
        {
            "attributes": {
                "OBJECTID": 1,
                "Description": "This is a test feature."
            },
            "geometry": {
                "x": -123.80816799999991,
                "y": 39.40451500000006
            }
        },
        {
            "attributes": {
                "OBJECTID": 2,
                "Description": "This is aanother test feature."
            },
            "geometry": {
                "x": -123.80816839299996,
                "y": 39.404514814000095
            }
        }
    ]
}"""

# convert to RecordSet object (should be able to use this most places FeatureSet would be called for)
record_set = arcpy.AsShape(valid_featureset_json, True)

# If you need a FeatureClass, just copy into one
output_fc = r'in_memory\\output_fc'
arcpy.management.CopyFeatures(record_set, output_fc)

And that’s it! No need to write the json to file.

Fun with Python Sets (and arcpy)

Thu, 21 Jul 2016 17:38:05 -0400

I often forget about the Python set, so this is a fun reminder to myself to keep it in mind. Here’s a neat example of using a set for deleting fields.

This function deletes all of the fields except the ones you specify:

def delete_all_fields_except(fc, keep_fields):
    """ deletes all fields except those specfied as keep_fields """ 
    # create set of all possible fields but exclude OBJECTID, SHAPE, etc.
    all_fields = set(f.name for f in arcpy.ListFields(fc) if not f.required)

    # get a list of fields in the featureclass but not in the list of fields to keep
    # using set.difference()
    
    delete_fields = list(all_fields.difference(keep_fields))
    
    # delete out all the fields
    arcpy.management.DeleteField(fc, delete_fields)

Usage:

# Assume our feature class has the following fields:
print([f.name for f in arcpy.ListFields(fc)])
# ['OBJECTID', 'Shape', 'field1', 'field2', 'field3', 'field4', 'field5']

keep_fields = ['field1', 'field3']
delete_all_fields_except(fc, keep_fields)

print([f.name for f in arcpy.ListFields(fc)])
# ['OBJECTID', 'Shape', 'field1', 'field3']

This is pretty simple example and you could obviously accomplish this without using set, but set is more elegant and more efficient (probably doesn’t matter much here). Perhaps I’ll update this post with more advanced usage now that this is here to remind me to think about sets!

arcpy: Efficient Geometry Creation

Thu, 04 Feb 2016 15:10:09 -0500

Sometimes every second counts… and even if it doesn’t, it’s still interesting to see the quirks of a familiar library.

It turns out that object creation can be somewhat expensive (especially when you’re talking about Python –> ArcObjects –> COM). With arcpy (and underlying ArcObjects), there are some objects which can be reused to gain some efficiency.

An interesting example is simply creating a polyline from a pair of points.

Example: Creating a 2 point polyline

Simplest form:

In its simplest form, you may write:

def create_line_simple(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    start_point = arcpy.Point(*point1_x_and_y)
    end_point = arcpy.Point(*point2_x_and_y)
    array = arcpy.Array([start_point, end_point])
    polyline = arcpy.Polyline(array, spatial_ref)
    return polyline

# Usage:
polyline = create_line_simple((-81.60, 36.20), (-81.70, 36.30), arcpy.SpatialReference(4326))

There’s nothing wrong with this code. In fact, if you’re only creating a few polylines, stop here. It’s readable and gets the job done.

A little more efficient:

However, if you’re creating thousands of polylines, some time can be saved by reusing arcpy.Point objects.

# modules scoped private variables
_start_point = arcpy.Point()
_end_point = arcpy.Point()

def create_line_reuse_points(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    _start_point.X, _start_point.Y = point1_x_and_y
    _end_point.X, _end_point.Y = point2_x_and_y
    array = arcpy.Array([_start_point, _end_point])
    polyline = arcpy.Polyline(array, spatial_ref)
    return polyline

In this case, we’re creating 2 module scoped points only once and then setting the X and Y properties on those points. The arcpy.Polyline constructor reads X and Y from those points, but it doesn’t maintain a reference to the points. Setting properties on the existing objects is a bit more efficient than creating new objects every time and since references aren’t maintained to those objects, we’re safe from a memory perspective.

Even more efficient:

Why not go ahead and reuse the arcpy.Array as well? Once again, arcpy.Polyline() only reads data from the array and doesn’t maintain a reference. Make sure to removeAll() from array to clean up.

# modules scoped private variables
_start_point = arcpy.Point()
_end_point = arcpy.Point()
_array = arcpy.Array()

def create_line_reuse_points_array(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    _start_point.X, _start_point.Y = point1_x_and_y
    _end_point.X, _end_point.Y = point2_x_and_y
    _array.add(_start_point)
    _array.add(_end_point)
    polyline = arcpy.Polyline(_array, spatial_ref)
    _array.removeAll()
    return polyline

How much more efficient is this approach?

Here are the results (in seconds) for creating 100,000 polylines with each function (Python 3.4.1 w/ ArcGIS Pro on Core i7-4712HQ):

Create line simple:
0:00:21.071529

Create line reuse points:
0:00:17.813275

Create line reuse points and array:
0:00:16.277035

Is it a huge difference? Not really. But if you have a process that creates a large amount of geometries, it’s worth considering reusing a few objects.

Here’s the full test script to produce the above results:

import arcpy
from datetime import datetime as dt


def time_me(n):
    """ decorator to print total time to run function n number of times """
    def time_me_decorator(f):
        def wrapper(*args):
            start = dt.now()
            for _ in range(n):
                f(*args)
            print(dt.now() - start)
        return wrapper
    return time_me_decorator


REPETITIONS = 100000

######## Simple Case

@time_me(REPETITIONS)
def create_line_simple(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    start_point = arcpy.Point(*point1_x_and_y)
    end_point = arcpy.Point(*point2_x_and_y)
    array = arcpy.Array([start_point, end_point])
    polyline = arcpy.Polyline(array, spatial_ref)
    return polyline


######## Reuses the point objects

# modules scoped private functions
_start_point = arcpy.Point()
_end_point = arcpy.Point()

@time_me(REPETITIONS)
def create_line_reuse_points(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    _start_point.X, _start_point.Y = point1_x_and_y
    _end_point.X, _end_point.Y = point2_x_and_y
    array = arcpy.Array([_start_point, _end_point])
    polyline = arcpy.Polyline(array, spatial_ref)
    return polyline


######## Reuses the point and array objects

# modules scoped private functions
_start_point = arcpy.Point()
_end_point = arcpy.Point()
_array = arcpy.Array()

@time_me(REPETITIONS)
def create_line_reuse_points_array(point1_x_and_y, point2_x_and_y, spatial_ref):
    """ creates polyline from pair of (x,y) tuples """
    _start_point.X, _start_point.Y = point1_x_and_y
    _end_point.X, _end_point.Y = point2_x_and_y
    _array.add(_start_point)
    _array.add(_end_point)
    polyline = arcpy.Polyline(_array, spatial_ref)
    _array.removeAll()
    return polyline


# Run our tests
if __name__ == "__main__":
    WGS_84 = arcpy.SpatialReference(4326)
    POINT1 = (-81.674525, 36.216630)
    POINT2 = (-81.675351, 36.213886)

    print("Create line simple:")
    create_line_simple(POINT1, POINT2, WGS_84)
    print("")

    print("Create line reuse points:")
    create_line_reuse_points(POINT1, POINT2, WGS_84)
    print("")

    print("Create line reuse points and array:")
    create_line_reuse_points_array(POINT1, POINT2, WGS_84)

arcpy.Project in_memory Featureclass

Thu, 10 Sep 2015 00:00:00 +0000

It’s inevitable that you eventually run into this error when scripting with arcpy (arcpy.Project_management): https://help.arcgis.com/en%20/arcgisdesktop/10.0/help/index.html#//00vp0000000m000944.htm

The standard project tool does not support in_memory workspaces.

Here’s the workaround - we just create a new featureclass using the source featureclass as a template and then exploit the spatial_reference parameter of arcpy.da.SearchCursor to project on the fly while inserting into the new featureclass.

Function:

from os.path import split
import arcpy

# create destination feature class using the source as a template to establish schema
# and set destination spatial reference
def project(source_fc, out_projected_fc, spatial_reference):
  """ projects source_fc to out_projected_fc using cursors (supports in_memory workspace) """
  path, name = split(out_projected_fc)
  arcpy.management.CreateFeatureclass(path, name,
                                      arcpy.Describe(source_fc).shapeType,
                                      template=source_fc,
                                      spatial_reference=spatial_reference)

  # specify copy of all fields from source to destination
  fields = ["Shape@"] + [f.name for f in arcpy.ListFields(source_fc) if not f.required]

  # project source geometries on the fly while inserting to destination featureclass
  with arcpy.da.SearchCursor(source_fc, fields, spatial_reference=spatial_reference) as source_curs, \
       arcpy.da.InsertCursor(out_projected_fc, fields) as ins_curs:
      for row in source_curs:
        ins_curs.insertRow(row)

Usage:

# assume we've already created this somewhere
source_fc = r"in_memory/source_fc"

# destination featureclass to be created
out_projected_fc = r"in_memory/projected_source_fc"

# destination projection
web_mercator = arcpy.SpatialReference(102100)

project(source_fc, out_projected_fc, web_mercator)

Edit 2016-02-04: Changed to function per suggestion by Andy Garfield

arcpy.da Cursors and Readability Part 2

Sun, 21 Dec 2014 00:00:00 +0000

arcpy.da cursors provide the simplest data structure by default (tuples). Python generators provide a pretty neat way of customizing these cursors to increase readability.

SearchCursor:

Return dict instead of tuple:

def rows_as_dicts(cursor):
  """ returns rows as dicts (does not maintain field order) """
  colnames = cursor.fields
  for row in cursor:
    yield dict(zip(colnames, row))

Usage (note the cursor is wrapped with the generator):

total_cost_by_name = {}
with arcpy.da.SearchCursor(costs_table, ["name", "unit_cost", "quantity"]) as curs:
  for row in rows_as_dicts(curs):
    total_cost_by_name[row["name"]] = row["unit_cost"] * row["quantity"]

If using with an update cursor, you’ll want to use an OrderedDict so the field/value order is maintained. You’ll also need to the use dict.values() to pass an ordered tuple back into Cursor.updateRow.

UpdateCursor:

Return collections.OrderedDict instead of tuple:

import collections
def rows_as_ordered_dicts(cursor):
  """ returns rows as collections.OrderedDict """
  colnames = cursor.fields
  for row in cursor:
    yield collections.OrderedDict(zip(colnames, row))

Usage (updating):

with arcpy.da.UpdateCursor(costs_table, ["unit_cost", "quantity", "total_cost"]) as curs:
  for row in rows_as_ordered_dicts(curs):
    row["total_cost"] = row["unit_cost"] * row["quantity"]

    # call .values() to convert back to tuple
    # if we didn't use OrderedDict, the values would be in random order.
    curs.updateRow( row.values() )

arcpy.da Cursors and Readability Part 1

Thu, 07 Aug 2014 00:00:00 +0000

The arcpy.da cursors (ie: arcpy.da.SearchCursor) are far and away better than the regular cursors (ie: arcpy.SearchCursor). They’re more pythonic, incredibly flexible, faster, and allow context management (with). However, their straight-out-of-the-box-use may be a little hard to read and maintain.

Fun with Tuples

arcpy.da cursors return tuples (instead of objects that feel too much like writing ArcObjects).

Good:

It’s perfectly valid to use tuple indexes to access the fields:

total_cost_by_name = {}
with arcpy.da.SearchCursor(costs_table, ["name", "unit_cost", "quantity"]) as curs:
    for row in curs:
        total_cost_by_name[row[0]] = row[1] * row[2]

Better:

This works fine, but the intent of our calculation is not really clear without some mental mapping. How about this?

total_cost_by_name = {}
with arcpy.da.SearchCursor(costs_table, ["name", "unit_cost", "quantity"]) as curs:
    for row in curs:
        name = row[0]
        unit_cost = row[1]
        quantity = row[2]

        total_cost_by_name[name] = unit_cost * quantity

The intent in the calculation is much more clear here, but we’ve gained a lot of verbosity that is perhaps unnecessary.

Best:

Of course, you could set the variables all on one line name, unit_cost, quantity = row[0], row[1], row[2], but even better would be to take advantage of python’s tuple unpacking: name, unit_cost, quantity = row. Taking that one step further, why not unpack every tuple as it’s iterated over. Now we have a process that is both concise and highly readable:

total_cost_by_name = {}
with arcpy.da.SearchCursor(costs_table, ["name", "unit_cost", "quantity"]) as curs:
    for name, unit_cost, quantity in curs:
        total_cost_by_name[name] = unit_cost * quantity

Now what if we want to update a total cost field in the featureclass using an UpdateCursor instead?

with arcpy.da.UpdateCursor(costs_table, ["unit_cost", "quantity", "total_cost"]) as curs:
    for unit_cost, quantity, total_cost in curs:
        total_cost = unit_cost * quantity
        curs.updateRow( (unit_cost, quantity, total_cost) )

arcpy: Reorder Fields

Thu, 17 Apr 2014 00:00:00 +0000

Here’s another handy function for re-ordering fields in a feature class. Like the rename_fields function previously posted, it recreates the existing field mappings and modifies as necessary. The output is a new feature class with fields in the order specified.

The only caveat is that required fields always get pushed to the front (so the featureclass will start with OBJECTID, Shape, rest of fields…). After required fields are out of the way, the rest of the fields are added in the order specified in the field_order list. If any fields are missing, they are added to the end of the feature class.

import arcpy

def reorder_fields(table, out_table, field_order, add_missing=True):
    """
    Reorders fields in input featureclass/table
    :table:         input table (fc, table, layer, etc)
    :out_table:     output table (fc, table, layer, etc)
    :field_order:   order of fields (objectid, shape not necessary)
    :add_missing:   add missing fields to end if True (leave out if False)
    -> path to output table
    """
    existing_fields = arcpy.ListFields(table)
    existing_field_names = [field.name for field in existing_fields]

    existing_mapping = arcpy.FieldMappings()
    existing_mapping.addTable(table)

    new_mapping = arcpy.FieldMappings()

    def add_mapping(field_name):
        mapping_index = existing_mapping.findFieldMapIndex(field_name)

        # required fields (OBJECTID, etc) will not be in existing mappings
        # they are added automatically
        if mapping_index != -1:
            field_map = existing_mapping.fieldMappings[mapping_index]
            new_mapping.addFieldMap(field_map)

    # add user fields from field_order
    for field_name in field_order:
        if field_name not in existing_field_names:
            raise Exception("Field: {0} not in {1}".format(field_name, table))

        add_mapping(field_name)

    # add missing fields at end
    if add_missing:
        missing_fields = [f for f in existing_field_names if f not in field_order]
        for field_name in missing_fields:
            add_mapping(field_name)

    # use merge with single input just to use new field_mappings
    arcpy.Merge_management(table, out_table, new_mapping)
    return out_table

Usage:

new_field_order = ["field2", "field3", "field1"]
reorder_fields(in_fc, out_fc, new_field_order)

I’ve tested this in 10.1.1+ - it may work in 10.0 as well and should work with any license level.

arcpy: Rename Fields

Tue, 01 Apr 2014 07:43:15 -0400

In ArcMap 10.2.1, esri finally added a tool to allow renaming of fields (Data Management -> Alter Field).

This works well for a single field, but you may need something different if:

you’re still on 10.0/10.1,
you want to rename several fields at once,
you would prefer not to rename the fields in-place,
you need to rename fields in a source unsupported by the Alter Field tool (hint: shapefiles).

Here’s a great function for renaming fields (and maintaining their order). It can be used in the python console.

def rename_fields(table, out_table, new_name_by_old_name):
    """ Renames specified fields in input feature class/table
    :table:                 input table (fc, table, layer, etc)
    :out_table:             output table (fc, table, layer, etc)
    :new_name_by_old_name:  {'old_field_name':'new_field_name',...}
    ->  out_table
    """
    existing_field_names = [field.name for field in arcpy.ListFields(table)]

    field_mappings = arcpy.FieldMappings()
    field_mappings.addTable(table)

    for old_field_name, new_field_name in new_name_by_old_name.iteritems():
        if old_field_name not in existing_field_names:
            message = "Field: {0} not in {1}".format(old_field_name, table)
            raise Exception(message)

        mapping_index = field_mappings.findFieldMapIndex(old_field_name)
        field_map = field_mappings.fieldMappings[mapping_index]
        output_field = field_map.outputField
        output_field.name = new_field_name
        output_field.aliasName = new_field_name
        field_map.outputField = output_field
        field_mappings.replaceFieldMap(mapping_index, field_map)

    # use merge with single input just to use new field_mappings
    arcpy.Merge_management(table, out_table, field_mappings)
    return out_table

The function accepts the input feature class, path to output, and a dictionary mapping the old names to the new names:

Usage:

# does need need to include all fields, only those you want to rename
new_name_by_old_name = { 'old_name_1':'new_name_1',
                         'old_name_2':'new_name_2' }
rename_fields(in_fc, renamed_fc, new_name_by_old_name)

This works by duplicating the arcpy.FieldMappings object from the existing feature class, renaming the fields within the mapping based on the input, and then calling a function that will apply that mapping. Merge with a single input gets the job done. Using the field mappings is a nice trick for efficiently managing fields.