Skip to main content

EAAMO Reflections: Caleb

RDS@Pitt posts on events

By Caleb Finamore

Editor's Note: this is the final part in our mini-series on the EAAMO conference.
Part 1 | Part 2

Last month, Pitt hosted the 2025 ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. Over the course of the week, scholars from all over met with the goal of advancing understanding of both the societal embedding of technology as well as how techniques from algorithms, optimization, and mechanism design can help improve equity and access to opportunity for underserved communities through the lenses of both the social sciences and humanistic studies.

I sat in on the first day of the Principles and Performance Workshop, serving as a note-taker for and contributor to the Public Sector AI Procurement breakout group. Our task was to review, discuss and ask questions regarding the AI Factsheet for Third Party Systems, a proposal from City Detect, Inc. to the city of San Jose, CA. In turn, a committee would then review our review, using our discussion to discuss larger frameworks surrounding the procurement of AI products for governmental, commercial, and educational purposes. The way I describe the process makes it sound a little absurd, but I think the reality is that most things we do are a little absurd. I think that’s ok.

Absurdities in mind, the breakout group was an endless wealth of discussion, the pipe dream of every undergraduate professor who has ever stared into the void of sixty-some laminated eyes, forced to ask the dreaded question, “so who did the reading?” I arrived prepared to speak, but I did not expect to, and I did not have to. Aside from occasionally googling facts, figures, and the million-dollar brand identities of Texas cities, my role was to listen to and record the conversation. To sit, to look at people looking at other people, to clatter on my old, gunked-up keyboard until it let me type the letter p, to smile politely in a way which says "I know you didn't mean to do that, it's okay to look away" whenever anyone accidentally made eye contact with me. Following are my unedited notes on the session, left unedited as recognition of my role in this sphere: not leading, not participating, but transcribing as transparently as possible. I would like to do it again sometime.

  1. Began by raising issues:
    1. People won’t be the biggest fan of installing a bunch of cameras to be surveilled
      1. What about using existing cameras?
      2. Treat it like a police style model: must demonstrate exhaustion of lower level resources before escalation
    2. Can you do anything about it regardless?
      1. Detroit identified a lot of blight, but hasn’t reached goal of dealing with it
  2. Moved to addressing quality and ownership of data
    1. State data must be public, but private corporation data can be private, so where would the data for this be stored?
    2. However, even public data can be redacted if it’s used in connection to private stakeholders
  3. Raised concern with evaluation metrics disclosed by City Detect
    1. “F1-score” is not specific, and the lack of specificity is concerning given that 85% is not a particularly high bar
    2. If you were a potential client, what metrics would you be looking for? Would that change based on your background?
    3. Concern about order of operations: soliciting RFPs to solve blight problem vs being solicited to by corporations
      1. First mover advantage: “hey, I’m corporation X, here’s how I solve this problem”
      2. Political pressure when being solicited to to look like you were forward thinking as a government 
  4. What metrics for accountability exist and are presented? 
    1. What metrics exist within AI at large?
      1. Not a metric per se, but making sure the people who are receiving reports and outputs have the technical skills to parse the output they’re receiving 
      2. Comprehensive audit: testing inputs and outputs, looking for patterns
    2. Who is the burden of proof on?
    3. Will the model present proof of the blight? Will citizens have means to contest the blight?
    4. What happens if the model is wrong? Who bears the financial responsibility?
    5. Answers to above questions will vary heavily depending on local legislation and procedures
    6. Expressed concern about City Detect’s human review
      1. human review cannot be comprehensive, because if it was, you could just employ the human, so if the review isn’t comprehensive, is it good?
  5. Raised concern that this feels like a way to exacerbate economic inequalities
    1. People who will have blights won’t have the money to fix them, so it could easily turn into liens and compounded fines
  6. Agreed would like to see a more detailed procedure for use cases of model
  7. Opened discussion of speed at which they deploy new models
    1. Why so many models?
    2. Why is the rate so fast?

OUT OF TIME