This sort of experience shows how broken the FOIA law is. If it’s in the public interest to make data available, it’s in the public interest to make it available to a person with imperfect understanding of the extreme details of government’s crappy IT systems.
Not sure exactly what the fix is, but one idea is to have a state-wide ombudsman-like office for facilitating FOIA requests. Currently each agency usually has its own small FOIA office, which naturally protects its own turf. A centralized office could
1. …be independent of the agencies from which info is being requested, avoiding conflicts of interest in denying/delaying requests
2. …have commitments to confidentiality so agencies couldn’t justify withholding contextual info (“what’s a better way to ask this question?”) from the ombudsman
3. …afford building up more technical and legal expertise than any single agency-specific office.
I don't think FOIA is broken, it just needs some tweaks. One of the tweaks OP is involved with is changing the FOIA law to allow for the type of request he made. This is the only way to fix the poorly-decided Ill. S. Ct. case now.
There does need to be a better incentive for public bodies to do the right thing, though. I get a ton of weak, half-baked responses, and a ton of push-back. A lot of it feels like laziness. There is no decent punishment in Illinois for workers or bodies that don't make a reasonable effort to respond.
Tweaks like you describe are exactly what I think are doomed. The problem isn’t some small wording in the law that we just need to adjust. The problem is that (1) info request of complicated systems by their nature require good-faith coordination by both inquirer and respondent but (2) the agency responding to the FAOI request is incentivized to obstruct. This is why legal discovery is such a nightmare and so inefficient.
“Punish people for not being helpful enough” is a bad way to get them to be helpful.
Why aren't all non-classified, non-sensitive public data actually public by default? The time, effort and money they spent fighting the FOIA lawsuit - wouldn't it just be easier and cheaper to just honor the request?
Providing public APIs to such information sounds great and I think it should be done but is also probably realistically expensive and peone to security problems. The government seems to have a very difficult time in general running all sorts of web services, it's a nightmare in some jurisdictions to even pay the government money for e.g. tickets... I had a hell of a time paying a ticket in San Diego and had to visit the the courthouse multiple times and file a paper form because my paymentment became "late" when it never showed up in their online database weeks later. I was TRYING to pay them and even going to suffer through the "convenience" fees because they outsourced the website to some crappy company and they still couldn't get it right.
As I understand it, the trial was about whether or not providing the database schema is a security risk.
> Your request seeks a copy of tables or columns within each table of CANVAS. The dissemination of these pieces of network information could jeopardize the security of the systems of the City of Chicago.
The first reasonable concern that springs to mind is that correlating a few fields that are individually non-identifying in a dataset can lead to deanonymizing people; in principle, the FOIA process gives the organization being requested time to think about what needs to be masked to protect privacy.
If government has access to all those fields the data isn't private.
Your argument makes me think we really SHOULD make all government data public. Then people would have an incentive to not let governments have so much of their private data.
> in principle, the FOIA process gives the organization being requested time to think about what needs to be masked to protect privacy.
Although of course that's a lot more in principle than in practice! Like rolling your own crypto, I think experience shows that, against determined de-anonymizers, there is basically nothing you can do to preserve anonymity except to severely limit the information, and the only way definite feedback you get is if you don't succeed and someone discloses the de-anonymized information that they were able to reconstruct.
Determining whether or not something is sensitive is not necessarily a trivial activity, especially if you’re asking it to be done for everything that every employee creates everyday.
Thats why you do design o DB and decide if this data is really needed to be stored, for how long, now sensitive it is etc. You don't just add data to DB as you go (well ideally).
If only it were that easy. Only a small amount of government data is in a database, if even electronic at all. A lot of it is on paper, in Microsoft word, in emails, in a voicemail, etc.
Nobody is going to write a DB schema or call a DBA before taking notes in a notebook… but that’s government data too.
Because it would significantly de-legitimize the government (particularly state and local) if any and every youtube talking head and blogger could pour over it and find the objectionable things that is supported by few and only tolerated by many because of poor awareness. And that's before you get into the spurious correlations and conspiracy stuff.
I'm not really interested in any of that. But I am interested in us having privacy from government. The last thing I want is someone like Musk getting access to my personal information, and the only way to really prevent that is to not collect it in the first place.
I wonder if starting with intentionally getting a parking fine in Chicago, followed by then submitting an FOIA about that fine and all related documents / data would have worked.
So dumb that the default behavior of the governments (State and Fed) is to withhold information. This isn't classified information and shouldn't be treated as such.
I reiterate my point from the comments of the companion post. OP lost even while being represented by some of the best civil rights lawyers in the country.
A lot of FOIA requests die because they receive push-back and the requestor lacks the resources to litigate it. You can do it yourself. FOIA litigation is usually not like OP's struggle over data types -- it's usually just to get the court to smack the public body and tell them they are being lazy or overly strict and the court procedures are much simpler. (often the public body will fold as soon as you file)
Also, I wonder if @chaps can give his reasoning on going directly for litigation? In Illinois there is an alternate avenue where you can ask the AG to intervene. (I hate this route myself because it has become slow and toothless)
Going to litigation made sense here because, tbh, I didn't want to deal with the PAC's office taking three years to complete the RFR while never actually understanding the underlying issue. They really stink at interpreting anything technical. Also, I've never done pro bono (I probably could though), but I don't trust myself on procedural matters!
> Also, I wonder if @chaps can give his reasoning on going directly for litigation? In Illinois there is an alternate avenue where you can ask the AG to intervene. (I hate this route myself because it has become slow and toothless)
There is some discussion here, which you may already have seen:
This sort of experience shows how broken the FOIA law is. If it’s in the public interest to make data available, it’s in the public interest to make it available to a person with imperfect understanding of the extreme details of government’s crappy IT systems.
Not sure exactly what the fix is, but one idea is to have a state-wide ombudsman-like office for facilitating FOIA requests. Currently each agency usually has its own small FOIA office, which naturally protects its own turf. A centralized office could 1. …be independent of the agencies from which info is being requested, avoiding conflicts of interest in denying/delaying requests 2. …have commitments to confidentiality so agencies couldn’t justify withholding contextual info (“what’s a better way to ask this question?”) from the ombudsman 3. …afford building up more technical and legal expertise than any single agency-specific office.
I don't think FOIA is broken, it just needs some tweaks. One of the tweaks OP is involved with is changing the FOIA law to allow for the type of request he made. This is the only way to fix the poorly-decided Ill. S. Ct. case now.
There does need to be a better incentive for public bodies to do the right thing, though. I get a ton of weak, half-baked responses, and a ton of push-back. A lot of it feels like laziness. There is no decent punishment in Illinois for workers or bodies that don't make a reasonable effort to respond.
This is basically how FOIA responses go:
https://www.youtube.com/watch?v=BxWQo_vZgR8
Tweaks like you describe are exactly what I think are doomed. The problem isn’t some small wording in the law that we just need to adjust. The problem is that (1) info request of complicated systems by their nature require good-faith coordination by both inquirer and respondent but (2) the agency responding to the FAOI request is incentivized to obstruct. This is why legal discovery is such a nightmare and so inefficient.
“Punish people for not being helpful enough” is a bad way to get them to be helpful.
>“Punish people for not being helpful enough” is a bad way to get them to be helpful.
That's the reality every vendor rep's job exists in and most of them are quite helpful.
Why aren't all non-classified, non-sensitive public data actually public by default? The time, effort and money they spent fighting the FOIA lawsuit - wouldn't it just be easier and cheaper to just honor the request?
Providing public APIs to such information sounds great and I think it should be done but is also probably realistically expensive and peone to security problems. The government seems to have a very difficult time in general running all sorts of web services, it's a nightmare in some jurisdictions to even pay the government money for e.g. tickets... I had a hell of a time paying a ticket in San Diego and had to visit the the courthouse multiple times and file a paper form because my paymentment became "late" when it never showed up in their online database weeks later. I was TRYING to pay them and even going to suffer through the "convenience" fees because they outsourced the website to some crappy company and they still couldn't get it right.
Expensive compared to what? Paying a bunch of lawyers and dragging a lawsuit for five years? Don’t lawyers, judges, court personnel’s time cost money?
They even refused to provide the database schema - how is providing just the schema expensive? Or hard to do?
As I understand it, the trial was about whether or not providing the database schema is a security risk.
> Your request seeks a copy of tables or columns within each table of CANVAS. The dissemination of these pieces of network information could jeopardize the security of the systems of the City of Chicago.
Related: https://news.ycombinator.com/item?id=43175628
Nobody said anything about providing a public API.
Just provide a DB dump to this one guy.
The first reasonable concern that springs to mind is that correlating a few fields that are individually non-identifying in a dataset can lead to deanonymizing people; in principle, the FOIA process gives the organization being requested time to think about what needs to be masked to protect privacy.
If government has access to all those fields the data isn't private.
Your argument makes me think we really SHOULD make all government data public. Then people would have an incentive to not let governments have so much of their private data.
> in principle, the FOIA process gives the organization being requested time to think about what needs to be masked to protect privacy.
Although of course that's a lot more in principle than in practice! Like rolling your own crypto, I think experience shows that, against determined de-anonymizers, there is basically nothing you can do to preserve anonymity except to severely limit the information, and the only way definite feedback you get is if you don't succeed and someone discloses the de-anonymized information that they were able to reconstruct.
Determining whether or not something is sensitive is not necessarily a trivial activity, especially if you’re asking it to be done for everything that every employee creates everyday.
Thats why you do design o DB and decide if this data is really needed to be stored, for how long, now sensitive it is etc. You don't just add data to DB as you go (well ideally).
If only it were that easy. Only a small amount of government data is in a database, if even electronic at all. A lot of it is on paper, in Microsoft word, in emails, in a voicemail, etc.
Nobody is going to write a DB schema or call a DBA before taking notes in a notebook… but that’s government data too.
Example: https://home.nps.gov/katm/blogs/images/IMG_3229.jpg
True, but in this case we are talking about providing DB dumps to citizens.
That is not how I interpreted “all non-classified, non-sensitive public data”
true. i scope of grand parent comment. i was still thinking about TFA when i commented.
Because it would significantly de-legitimize the government (particularly state and local) if any and every youtube talking head and blogger could pour over it and find the objectionable things that is supported by few and only tolerated by many because of poor awareness. And that's before you get into the spurious correlations and conspiracy stuff.
Or it would make us look for ways to have government do fewer things.
About as believable as DOGE "saving" the government $105 billion.
I'm not really interested in any of that. But I am interested in us having privacy from government. The last thing I want is someone like Musk getting access to my personal information, and the only way to really prevent that is to not collect it in the first place.
Recent and related:
I Went to SQL Injection Court - https://news.ycombinator.com/item?id=43175628 - Feb 2025 (433 comments)
I wonder if starting with intentionally getting a parking fine in Chicago, followed by then submitting an FOIA about that fine and all related documents / data would have worked.
Edit: seems like that was the part of the origin story of this according to https://sockpuppet.org/blog/2025/02/09/fixing-illinois-foia/
So dumb that the default behavior of the governments (State and Fed) is to withhold information. This isn't classified information and shouldn't be treated as such.
100% onboard with shrinking the government.
I reiterate my point from the comments of the companion post. OP lost even while being represented by some of the best civil rights lawyers in the country.
A lot of FOIA requests die because they receive push-back and the requestor lacks the resources to litigate it. You can do it yourself. FOIA litigation is usually not like OP's struggle over data types -- it's usually just to get the court to smack the public body and tell them they are being lazy or overly strict and the court procedures are much simpler. (often the public body will fold as soon as you file)
Also, I wonder if @chaps can give his reasoning on going directly for litigation? In Illinois there is an alternate avenue where you can ask the AG to intervene. (I hate this route myself because it has become slow and toothless)
Going to litigation made sense here because, tbh, I didn't want to deal with the PAC's office taking three years to complete the RFR while never actually understanding the underlying issue. They really stink at interpreting anything technical. Also, I've never done pro bono (I probably could though), but I don't trust myself on procedural matters!
> Also, I wonder if @chaps can give his reasoning on going directly for litigation? In Illinois there is an alternate avenue where you can ask the AG to intervene. (I hate this route myself because it has become slow and toothless)
There is some discussion here, which you may already have seen:
https://news.ycombinator.com/item?id=43176319
This government doesn't honor its commitments to friend, foes, or citizens
> Please note that in late 2013, the City of Chicago launched a publically available Data
did they really say "publically" in their response? :)