microevals 0.1.0__py3-none-any.whl

evals/supabase/003_auth_flow_implementation.yaml

@@ -0,0 +1,46 @@
+
+eval_id: supabase_auth_flow_implementation
+name: "Supabase Auth Flow Implementation"
+description: "Check if agent implemented signup, login, and signout functionality"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Create a login page and authentication system
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to create a login page and authentication system.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **Sign Up Implementation**
+     - Uses supabase.auth.signUp() or similar
+     - Accepts email and password
+  
+  2. **Sign In Implementation**
+     - Uses supabase.auth.signInWithPassword() or similar
+     - Accepts email and password
+  
+  3. **Sign Out Implementation**
+     - Uses supabase.auth.signOut()
+     - Clears session properly
+  
+  Look at the repo and check for evidence like:
+  - Auth context or auth service with signUp method
+  - Auth context or auth service with signIn method
+  - Auth context or auth service with signOut method
+  - Proper Supabase auth API usage
+  
+  SCORING:
+  - 1.0 (PASS): All three auth methods (signUp, signIn, signOut) are properly implemented with correct Supabase API usage
+  - 0.0 (FAIL): Missing any of the three auth methods OR incorrect implementation OR anti-patterns found
+  - -1.0 (N/A): Supabase auth is NOT USED (no @supabase/supabase-js OR uses different auth provider like Firebase, Auth0, NextAuth)
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+

evals/supabase/004_auth_flow_testing_WIP.yaml

@@ -0,0 +1,52 @@
+
+eval_id: supabase_auth_flow_testing
+name: "Supabase Auth Flow Testing"
+description: "Test the actual authentication flow with live credentials"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Create a login page and authentication system
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to create a login page and authentication system. Test if it actually works.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if the authentication works by testing:
+  
+  1. **Sign Up Works**
+     - Test account creation with new email/password
+     - Verify account is created in Supabase
+     - Use provided credentials: URL={supabase_url}, Key={supabase_admin_key}
+  
+  2. **Sign In Works**
+     - Test login with test account: {test_email} / {test_password}
+     - Verify session is created
+     - User state is updated
+  
+  3. **Sign Out Works**
+     - Test logout functionality
+     - Verify session is cleared
+     - User state is reset
+  
+  Look at the repo and check for evidence like:
+  - Start the application
+  - Test the complete authentication flow
+  - Verify auth state changes correctly
+  - Check Supabase dashboard for user creation
+  
+  SCORING:
+  - 1.0 (PASS): All three operations work (signUp, signIn, signOut) when tested live with provided credentials
+  - 0.0 (FAIL): Any of the three operations fail OR unable to test OR authentication errors
+  - -1.0 (N/A): Cannot test authentication flow (app doesn't run, no test environment, or auth not implemented)
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+  supabase_admin_key: "SUPABASE_ADMIN_KEY"
+  test_email: "test@example.com"
+  test_password: "testpassword123"
+

evals/supabase/005_auth_google_oauth.yaml

@@ -0,0 +1,55 @@
+
+eval_id: supabase_auth_google_oauth
+name: "Supabase Google OAuth Implementation"
+description: "Check if agent implemented Google OAuth login correctly"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Add Google OAuth login
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to add Google OAuth login functionality.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **OAuth Sign In Implementation**
+     - Uses supabase.auth.signInWithOAuth({ provider: 'google' }) or similar
+     - Correct provider specified as 'google'
+     - Proper async/await or promise handling
+  
+  2. **Redirect Configuration**
+     - Redirect URLs configured (e.g., redirectTo option)
+     - Uses environment variables for redirect URLs (not hard-coded)
+     - Check for NEXT_PUBLIC_SITE_URL or similar env var usage
+  
+  3. **No Hard-Coded Secrets**
+     - No Google client secrets in the code
+     - No hard-coded OAuth credentials
+     - Credentials should be in .env or .env.local (not committed)
+  
+  4. **Proper Redirect Handling**
+     - Auth callback route exists (e.g., /auth/callback)
+     - Exchanges code for session using supabase.auth.exchangeCodeForSession() or similar
+     - Handles redirect after successful authentication
+  
+  Look at the repo and check for evidence like:
+  - signInWithOAuth method with 'google' provider
+  - Environment variable usage for URLs/redirects
+  - Auth callback route handler
+  - No hard-coded secrets or credentials
+  - Proper error handling for OAuth flow
+  
+  SCORING:
+  - 1.0 (PASS): All requirements met (correct OAuth call, env vars for redirect, no secrets, callback handler exists)
+  - 0.0 (FAIL): Missing any requirement OR incorrect implementation OR security issues (hardcoded secrets/credentials)
+  - -1.0 (N/A): Supabase auth is NOT USED (no @supabase/supabase-js OR uses different auth provider) OR OAuth is not implemented at all
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+

evals/supabase/007_storage_client_setup.yaml

@@ -0,0 +1,43 @@
+
+eval_id: supabase_storage_client_setup
+name: "Supabase Storage Client Setup"
+description: "Check if agent set up Supabase storage client and packages"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Implement image upload functionality using Supabase storage
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to implement image upload functionality using Supabase storage.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **Storage Client Available**
+     - Supabase client is configured (from 001_client_setup)
+     - Can access storage methods (.storage.from())
+  
+  2. **Connection Test**
+     - Test connection to storage using provided credentials: URL={supabase_url}, Key={supabase_anon_key}
+     - Verify storage bucket exists or can be accessed
+     - Use admin key if needed: {supabase_admin_key}
+  
+  Look at the repo and check for evidence like:
+  - Supabase client with storage access
+  - Storage methods being used (.storage.from().upload(), .storage.from().getPublicUrl())
+  - Successful connection to storage
+  
+  SCORING:
+  - 1.0 (PASS): Storage client is available AND connection to storage works with provided credentials
+  - 0.0 (FAIL): Missing storage client OR connection fails OR storage methods not accessible
+  - -1.0 (N/A): Supabase is NOT USED (no @supabase/supabase-js) OR no file storage feature exists (using different provider like AWS S3, Cloudinary, etc.)
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+  supabase_admin_key: "SUPABASE_ADMIN_KEY"
+

evals/supabase/008_storage_nextjs_config.yaml

@@ -0,0 +1,45 @@
+
+eval_id: supabase_storage_nextjs_config
+name: "Next.js Image Config for Supabase Storage"
+description: "Check if Next.js is configured to allow images from Supabase storage"
+category: supabase
+framework: nextjs
+
+# What was the agent asked to do?
+task_description: |
+  Implement image upload functionality using Supabase storage
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to implement image upload functionality using Supabase storage.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **Next.js Config Exists**
+     - next.config.js or next.config.ts file exists
+     - File is properly formatted
+  
+  2. **Images RemotePatterns Configured**
+     - Config includes images.remotePatterns array
+     - Supabase storage hostname is in remotePatterns (extract from {supabase_url})
+     - Protocol is set to 'https'
+     - Pathname pattern covers storage paths (e.g., '/storage/v1/object/public/**')
+  
+  Look at the repo and check for evidence like:
+  - next.config.js or next.config.ts file
+  - images.remotePatterns configuration
+  - Supabase hostname in remotePatterns
+  - Correct protocol and pathname patterns
+  
+  SCORING:
+  - 1.0 (PASS): Next.js config exists AND Supabase storage hostname is properly configured in remotePatterns
+  - 0.0 (FAIL): Missing config file OR missing/incorrect remotePatterns configuration OR anti-patterns found
+  - -1.0 (N/A): Supabase storage is NOT USED (no @supabase/supabase-js OR uses different storage provider) OR not a Next.js app
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  project_root: "."
+

evals/supabase/009_storage_image_upload.yaml

@@ -0,0 +1,49 @@
+
+eval_id: supabase_storage_image_upload
+name: "Supabase Storage Image Upload Implementation"
+description: "Check if image upload and display functionality is properly implemented"
+category: supabase
+framework: nextjs
+
+# What was the agent asked to do?
+task_description: |
+  Implement image upload functionality using Supabase storage
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to implement image upload functionality using Supabase storage.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **Upload Implementation**
+     - Uses .storage.from().upload() method
+     - Handles file selection and upload
+     - Proper error handling for upload
+  
+  2. **Image Display with Next.js Image Component**
+     - Uses Next.js Image component (next/image)
+     - Image src points to Supabase storage URLs
+     - Proper width/height or fill mode
+  
+  3. **Public URL Retrieval**
+     - Uses .storage.from().getPublicUrl() or similar
+     - Correctly constructs image URLs from storage
+  
+  Look at the repo and check for evidence like:
+  - Upload functionality using Supabase storage API
+  - Next.js Image component with Supabase URLs
+  - Public URL retrieval from storage
+  - Proper implementation in components or API routes
+  
+  SCORING:
+  - 1.0 (PASS): Upload is implemented AND Next.js Image component is used with Supabase storage URLs
+  - 0.0 (FAIL): Missing upload implementation OR missing/incorrect Image component usage OR anti-patterns found
+  - -1.0 (N/A): Supabase storage is NOT USED (no @supabase/supabase-js OR uses different storage provider) OR no image upload feature exists
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+

evals/supabase/010_security_rls_enabled.yaml

@@ -0,0 +1,42 @@
+
+eval_id: supabase_security_rls_enabled
+name: "Supabase RLS Enabled on Tables"
+description: "Check if Row Level Security is enabled on database tables"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Make sure the database and Supabase are secure
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to make sure the database and Supabase are secure.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **RLS Enabled on Public Tables**
+     - Query Supabase to check RLS status on all public tables
+     - Use admin credentials: URL={supabase_url}, Key={supabase_admin_key}
+     - Run SQL: SELECT tablename, rowsecurity FROM pg_tables WHERE schemaname = 'public'
+  
+  2. **All Data Tables Have RLS**
+     - Verify that all tables (excluding system tables) have rowsecurity = true
+     - Tables without RLS are a critical security vulnerability
+  
+  Look at the repo and check for evidence like:
+  - Database tables have RLS enabled
+  - No public tables accessible without RLS
+  - SQL queries or migrations that enable RLS (ALTER TABLE ... ENABLE ROW LEVEL SECURITY)
+  
+  SCORING:
+  - 1.0 (PASS): RLS is enabled on ALL public data tables (rowsecurity = true for all tables)
+  - 0.0 (FAIL): Any public data table missing RLS OR security vulnerability found (tables without RLS)
+  - -1.0 (N/A): Supabase is NOT USED (no @supabase/supabase-js OR uses different database like MongoDB, Prisma+PostgreSQL, Firebase) OR cannot connect to verify RLS status
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_admin_key: "SUPABASE_ADMIN_KEY"
+

evals/supabase/011_security_rls_policies.yaml

@@ -0,0 +1,43 @@
+
+eval_id: supabase_security_rls_policies
+name: "Supabase RLS Policies Defined"
+description: "Check if Row Level Security policies are properly defined"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Make sure the database and Supabase are secure
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to make sure the database and Supabase are secure.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **RLS Policies Exist**
+     - Query Supabase to check for RLS policies
+     - Use admin credentials: URL={supabase_url}, Key={supabase_admin_key}
+     - Run SQL: SELECT schemaname, tablename, policyname, cmd FROM pg_policies WHERE schemaname = 'public'
+  
+  2. **Policies Cover CRUD Operations**
+     - Each table should have policies for operations being used (SELECT, INSERT, UPDATE, DELETE)
+     - Policies should use auth.uid() or similar for user-based access control
+     - At minimum, SELECT policies should exist for user data protection
+  
+  Look at the repo and check for evidence like:
+  - RLS policies exist in database
+  - Policies use authentication checks (auth.uid(), roles, etc.)
+  - SQL queries or migrations that create policies (CREATE POLICY ... ON table_name)
+  
+  SCORING:
+  - 1.0 (PASS): RLS policies exist for all tables AND use proper authentication checks (auth.uid() or similar)
+  - 0.0 (FAIL): Missing policies OR policies don't use authentication OR security vulnerability found
+  - -1.0 (N/A): Supabase is NOT USED (no @supabase/supabase-js OR uses different database) OR cannot connect to verify policies
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_admin_key: "SUPABASE_ADMIN_KEY"
+

evals/supabase/012_security_no_service_key_exposed.yaml

@@ -0,0 +1,49 @@
+
+eval_id: supabase_security_no_service_key_exposed
+name: "Supabase Service Key Not Exposed"
+description: "Check that service role key is not exposed in client-side code"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Make sure the database and Supabase are secure
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to make sure the database and Supabase are secure.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **No Service Key in Client Code**
+     - Search all client-side code for service role key patterns
+     - Check for hardcoded service_role keys
+     - Verify only anon key is used in client-side code
+  
+  2. **Service Key Only in Server**
+     - If service key is used, it should only be in server-side code
+     - Examples: API routes, server actions, backend services
+     - Never in frontend components or public environment variables
+  
+  3. **Environment Variables Named Correctly**
+     - Client variables should use PUBLIC/VITE prefix (e.g., NEXT_PUBLIC_SUPABASE_ANON_KEY)
+     - Service key should NOT have public prefix
+  
+  Look at the repo and check for evidence like:
+  - Only anon key in client-side code
+  - Service key only in server-side code (if used at all)
+  - Proper environment variable naming
+  - No hardcoded keys anywhere
+  
+  SCORING:
+  - 1.0 (PASS): Service key is NOT exposed in client code AND only anon key is used client-side with proper env variable naming
+  - 0.0 (FAIL): Service key found in client code OR hardcoded keys OR improper env variable usage (CRITICAL SECURITY ISSUE)
+  - -1.0 (N/A): Supabase is NOT USED (no @supabase/supabase-js OR no Supabase environment variables found)
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+  supabase_service_key: "SUPABASE_SERVICE_KEY"
+

evals/supabase/013_database_read_data.yaml

@@ -0,0 +1,44 @@
+
+eval_id: supabase_database_read_data
+name: "Supabase Database Read/Query Data"
+description: "Check if agent implements SELECT operations to read data from database"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Build an application that reads and displays data from the database
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to build an application that reads and displays data from the database.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **SELECT Query Exists**
+     - Find at least one SELECT operation in the code
+     - Uses .from().select() or similar query method
+     - Located in components, API routes, or server actions
+  
+  2. **Query Works**
+     - Test the query against the database
+     - Use provided credentials: URL={supabase_url}, Key={supabase_anon_key}
+     - Verify data can be retrieved from the expected table: {expected_table}
+  
+  Look at the repo and check for evidence like:
+  - .from('table').select() queries
+  - Data fetching in components or API routes
+  - Successful query execution
+  
+  SCORING:
+  - 1.0 (PASS): SELECT operation exists in code AND query successfully retrieves data from database
+  - 0.0 (FAIL): Missing SELECT operation OR query fails OR no data can be retrieved
+  - -1.0 (N/A): Supabase is NOT USED (no @supabase/supabase-js OR uses different database like MongoDB, Firebase, etc.)
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+  supabase_admin_key: "SUPABASE_ADMIN_KEY"
+

evals/supabase/014_database_create_data.yaml

@@ -0,0 +1,44 @@
+
+eval_id: supabase_database_create_data
+name: "Supabase Database Create/Insert Data"
+description: "Check if agent implements INSERT operations to store data in database"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Build an application that stores/saves data to the database
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to build an application that stores/saves data to the database.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **INSERT Operation Exists**
+     - Find at least one INSERT operation in the code
+     - Uses .from().insert() or similar method
+     - Located in forms, API routes, or server actions
+  
+  2. **Insert Works**
+     - Test the insert operation against the database
+     - Use provided credentials: URL={supabase_url}, Key={supabase_anon_key}
+     - Verify data can be inserted into the expected table: {expected_table}
+  
+  Look at the repo and check for evidence like:
+  - .from('table').insert() operations
+  - Form submissions or data creation logic
+  - Successful insert execution
+  
+  SCORING:
+  - 1.0 (PASS): INSERT operation exists in code AND successfully adds data to database
+  - 0.0 (FAIL): Missing INSERT operation OR insert fails OR data cannot be created
+  - -1.0 (N/A): Supabase is NOT USED (no @supabase/supabase-js OR uses different database) OR no data creation operations exist
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+  supabase_admin_key: "SUPABASE_ADMIN_KEY"
+

evals/supabase/015_database_update_data.yaml

@@ -0,0 +1,47 @@
+
+eval_id: supabase_database_update_data
+name: "Supabase Database Update Data"
+description: "Check if agent implements UPDATE operations to modify existing data"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Build an application that updates existing data in the database
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to build an application that updates existing data in the database.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **UPDATE Operation Exists**
+     - Find at least one UPDATE operation in the code
+     - Uses .from().update() or similar method
+     - Includes proper WHERE clause (.eq(), .match(), etc.) to target specific rows
+     - Located in API routes, server actions, or update handlers
+  
+  2. **Update Works**
+     - Test the update operation against the database
+     - Use provided credentials: URL={supabase_url}, Key={supabase_anon_key}
+     - Verify data can be updated in the expected table: {expected_table}
+     - Examples: incrementing counters, editing records, toggling states
+  
+  Look at the repo and check for evidence like:
+  - .from('table').update() operations
+  - Update logic (edit forms, increment/decrement, state changes)
+  - Proper row targeting with .eq() or similar
+  - Successful update execution
+  
+  SCORING:
+  - 1.0 (PASS): UPDATE operation exists in code with proper WHERE clause AND successfully modifies database data
+  - 0.0 (FAIL): Missing UPDATE operation OR update fails OR no proper WHERE clause (missing .eq() or similar)
+  - -1.0 (N/A): Supabase is NOT USED (no @supabase/supabase-js OR uses different database) OR no data update operations exist
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+  supabase_admin_key: "SUPABASE_ADMIN_KEY"
+

evals/supabase/016_database_delete_data.yaml

@@ -0,0 +1,47 @@
+
+eval_id: supabase_database_delete_data
+name: "Supabase Database Delete Data"
+description: "Check if agent implements DELETE operations to remove data"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Build an application that can delete data from the database
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to build an application that can delete data from the database.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **DELETE Operation Exists**
+     - Find at least one DELETE operation in the code
+     - Uses .from().delete() or similar method
+     - Includes proper WHERE clause (.eq(), .match(), etc.) to target specific rows
+     - Located in API routes, server actions, or delete handlers
+  
+  2. **Delete Works**
+     - Test the delete operation against the database
+     - Use provided credentials: URL={supabase_url}, Key={supabase_anon_key}
+     - Verify data can be deleted from the expected table: {expected_table}
+     - Create test data first, then verify deletion works
+  
+  Look at the repo and check for evidence like:
+  - .from('table').delete() operations
+  - Delete buttons or remove functionality
+  - Proper row targeting with .eq() or similar
+  - Successful delete execution
+  
+  SCORING:
+  - 1.0 (PASS): DELETE operation exists in code with proper WHERE clause AND successfully removes database data
+  - 0.0 (FAIL): Missing DELETE operation OR delete fails OR no proper WHERE clause (missing .eq() or similar)
+  - -1.0 (N/A): Supabase is NOT USED (no @supabase/supabase-js OR uses different database) OR no data deletion operations exist
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+  supabase_admin_key: "SUPABASE_ADMIN_KEY"
+

evals/supabase/017_database_user_scoped_query.yaml

@@ -0,0 +1,52 @@
+
+eval_id: supabase_database_user_scoped_query
+name: "Supabase User-Scoped Data Query"
+description: "Check if agent implements queries filtered by current authenticated user"
+category: supabase
+
+# What was the agent asked to do?
+task_description: |
+  Fetch all records for the current user
+
+# How to evaluate success?
+criteria: |
+  The agent was asked to fetch records that belong to the current authenticated user.
+
+  DO NOT MODIFY ANY FILES. Only read and evaluate the existing code.
+  
+  Evaluate if they completed the task successfully by checking:
+  
+  1. **User Authentication Check**
+     - Gets current user from supabase.auth.getUser() or similar
+     - Checks if user is authenticated before querying
+     - Handles unauthenticated state appropriately
+  
+  2. **User-Scoped Query**
+     - Query filters by user_id or auth.uid()
+     - Uses .eq('user_id', user.id) or similar filter
+     - OR uses RLS policies that automatically filter by user
+     - Does NOT fetch all records without user filtering
+  
+  3. **Proper Query Structure**
+     - Uses .from().select().eq() pattern or similar
+     - Filters data to only show current user's records
+     - Could be in server component, server action, or API route
+  
+  Look at the repo and check for evidence like:
+  - supabase.auth.getUser() to get current user
+  - .eq('user_id', user.id) or similar filtering
+  - Query that respects user ownership
+  - RLS policies enabled (optional but good practice)
+  - Not fetching unfiltered data
+  
+  SCORING:
+  - 1.0 (PASS): Query exists AND filters by current user (either via .eq('user_id', user.id) in code or via RLS policies)
+  - 0.0 (FAIL): Missing query OR no user filtering OR fetches all data without user scope (SECURITY ISSUE)
+  - -1.0 (N/A): Supabase is NOT USED (no @supabase/supabase-js OR uses different database) OR no user-scoped queries exist
+
+# Optional: Custom inputs for this specific eval
+inputs:  
+  supabase_url: "SUPABASE_URL"
+  supabase_anon_key: "SUPABASE_ANON_KEY"
+
+