New York City’s New AI Bias Law Is About To Take Effect: What You Need To Know

On July 5, 2023, the New York City Department of Consumer and Work Protection (DCWP) will begin enforcement of Local Law 144 (Law), which regulates employers’ use of “automated employment decision tools” (AEDTs) to screen applicants or employees in New York City when making employment decisions. AEDTs use machine learning, artificial intelligence (AI) and other processes to assist employers in evaluating candidates for hire or promotion. In an effort to curb the risk that AEDTs may produce biased results, the Law prohibits employers and employment agencies from using AEDTs unless certain bias audit, notice and reporting requirements are met. The Law also authorizes civil monetary penalties for violations.

This alert summarizes the key provisions.

What Is Local Law 144?

Local Law 144 makes it unlawful for New York City employers and employment agencies to use AEDTs to determine whether candidates should be selected or advanced in the hiring or promotion process unless (1) the AEDT is subject to a bias audit by an independent auditor before use and annually thereafter; (2) the results of the most recent bias audit and the AEDT’s distribution date are published on the employer’s or employment agency’s website; and (3) notice is provided to applicants and employees who are subject to automated screening at least 10 business days before use of the AEDT. The Law only applies when an employer or employment agency is screening candidates for employment or employees for promotion. Notably, the Law does not apply to terminations, work assignments or compensation adjustments.

Under the Law and accompanying regulations, any person who violates the Law is liable for a civil penalty of no more than $500 for the first violation (and each additional violation occurring on the same day of the first violation). Each subsequent violation will result in a civil penalty of at least $500 and not more than $1,500. Each failure to satisfy the applicant notice requirements constitutes a separate violation. Failure to meet the “bias audit” requirements results in separate, daily violations.

Notably, the Law is part of a larger nationwide trend regulating the use of AI in the employment space in an effort to prevent discrimination against job seekers and workers. For instance, on May 18, 2023, the U.S. Equal Employment Opportunity Commission (EEOC) released a technical assistance document discussing the agency’s views concerning the application of federal law to an employer’s use of automated systems, including those that incorporate AI. Similarly, on April 25, 2023, the EEOC joined officials from the DOJ, the FTC and the CFPB to release a joint statement pledging “to vigorously use [their] collective authorities to protect individuals’ rights regardless of whether legal violations occur through traditional means or advanced technologies,” such as AI.

What Is an AEDT?

The term “automated employment decision tool” means any computational process (i) derived from “machine learning, statistical modeling, data analytics, or artificial intelligence” that issues a “simplified output,” including a score, classification or recommendation, (ii) that is used to “substantially assist or replace” discretionary decision-making for making employment decisions that impact candidates.

The Law defines “machine learning, statistical modeling, data analytics, or artificial intelligence” as a group of mathematical, computer-based techniques:

• That generate a prediction, meaning an expected outcome for an observation, such as an assessment of a candidate’s fit or likelihood of success, or that generate a classification, meaning an assignment of an observation to a group, such as categorizations based on skill sets or aptitude; and
  
  
• Where a computer, at least in part, identifies the inputs, the relative importance placed on those inputs and, if applicable, other parameters for the models in order to improve the accuracy of the prediction or classification

An AEDT “substantially assist[s] or replace[s]” an employer’s discretionary decision-making when an employer or employment agency uses the AEDT and does any of the following:

• Relies solely on a simplified output generated by the AEDT, with no other factors considered
  
  
• Uses a simplified output as one of a set of criteria where the simplified output is weighted more than any other criterion in the set
  
  
• Uses a simplified output to overrule conclusions derived from other factors, including human decision-making

As used above, the term “simplified output” refers to the prediction or classification generated by the tool (e.g., score, tag, classification, ranking).

By way of example, use of an AEDT would be covered under the Law where the AEDT generates scores or ranks job applicants or employees (such as arranging a list of candidates based on how well their cover letters match the job description or rating a candidate’s technical skills) in the employment decision-making process, and where the AEDT-generated scores or ranks have more weight than either human discretion or any other factors.

Examples of tools that fall outside the scope of covered AEDTs include junk email filters, antivirus software, firewalls, calculators, spreadsheets, databases and other data compilations.

What Is a Bias Audit?

If an employer or employment agency is using an AEDT covered by the Law, then the Law requires it to conduct a “bias audit” by an “Independent Auditor” and publish a summary of the audit (and its distribution date) clearly and conspicuously before its use. A bias audit is a testing of an AEDT to assess the tool’s disparate impact on persons of specified demographics. Bias audits must be performed annually, and an AEDT cannot be used if more than one year has passed since the last bias audit of the AEDT was conducted.

An “Independent Auditor” is defined as objective individuals or groups who (i) are not and have not been involved in the use, development or distribution of the AEDT; (ii) have not at any point during the audit been employed by the employer, vendor or AEDT developer; and (iii) have no direct financial interest or material indirect financial interest in the use of the AEDT or the vendor that developed or distributed it.

The content of bias audits includes calculations of the “selection rate” for each category (or scoring rate, where the tool issues scores instead of classifications/groupings), including sex categories, race/ethnicity categories and intersectional categories, as well as the “impact ratio” of each category. A selection rate is the proportion of individuals in a certain category who were selected to move forward in the hiring process or assigned a classification by the tool being audited. The impact ratio measures the rate at which individuals in a particular group are selected as compared to the most selected group. These calculations are intended to reveal whether the tool is disproportionately affecting people of a particular category.

The Law also requires the publication of the number of individuals whose race and gender are “unknown” and who are thus excluded from the calculations. Additionally, an independent auditor may exclude a category that represents less than 2% of the data being used in the bias audit if the independent auditor justifies the exclusion.

An employer or employment agency should generally use its own “historical data” — meaning data collected during its use of an AEDT to assess candidates for employment or employees for promotion — when conducting a bias audit. An employer or employment agency using an AEDT may also use historical data from other employers or employment agencies that use the same AEDT to conduct the bias audit. Where insufficient historical data exists to conduct a statistically significant bias audit, an employer or employment agency may rely on a bias audit that uses test data. If test data is used, the summary of results of the bias audit should explain why historical data was not used and describe how the test data was generated and obtained.

Notice Requirements

The Law further requires employers and employment agencies to provide candidates for employment or promotion with notice 10 business days before any AEDT is used. The notice must include (i) that an AEDT is being used in assessing and evaluating the candidate; (ii) the job qualifications and characteristics the AEDT will use in its analysis; (iii) if not disclosed elsewhere on its website, the AEDT’s data source and type and the employer’s or employment agency’s data retention policy; and (iv) that a candidate may request an alternative selection process or accommodation. Such notice can be accomplished by a single clear notice on the employment section of the employer’s or employment agency’s website, in a job posting, or by way of U.S. mail or email to the candidate.

Who Is Subject to Local Law 144?

Employers and employment agencies located in New York City fall within the scope of Local Law 144. Further, with regard to the notice requirement, Local Law 144 appears to apply primarily to New York City residents who are “candidate[s] for employment” for a specific position in New York City, although neither the statute nor the regulation states this limitation clearly. A “candidate for employment” means a person who has applied for a specific employment position by submitting the necessary information or items in the format required by the employer or employment agency. Potential candidates who have not applied for a specific position are not covered.  

The DCWP held an informal discussion on May 22, 2023 and provided further guidance concerning the Law’s application:

• If the position is located within New York City, a bias audit is required along with notice for any New York City-residing candidates.
  
  
• If the position is outside New York City, a bias audit and notice are not required.
  
  
• Where the position is entirely remote, and the employer is solely located in New York City, a bias audit and notice for any candidates residing in New York City are required.
  
  
• Where the position is entirely remote, and the employer does not have an office in New York City, a bias audit and notice are not required.
  
  
• Where the position is entirely remote, but the employer has offices both in New York City and outside the city, a fact-specific analysis is required. Although not stated in the guidance, the analysis presumably would include whether the position involves contact and/or direct work with the New York City office, or the general presence of the employer in New York City. The “fact-specific” analysis is still in need of further clarification by the DCWP.

Next Steps and Best Practices

Employers and employment agencies that are using AI in employment decisions should take steps now to ascertain whether the tools they use constitute AEDTs ahead of Local Law 144’s July 5 enforcement date. Given the complexities surrounding the Law, companies should familiarize themselves with any AEDTs being used or considered in the workplace and ensure they are prepared to comply with the bias audit, notice and reporting requirements. As a best practice, employers and employment agencies should also keep records of (i) all electronic tools used to assist in hiring or promotion; (ii) how electronic tools were used and other methods that were used to evaluate candidates or employees; and (iii) all notices and methods of notice provided to candidates or employees.

Kramer Levin will continue to monitor developments in this area. Please contact a member of our team with questions or for assistance complying with Local Law 144.