diff --git a/docs/oracle.md b/docs/oracle.md new file mode 100644 index 0000000..ca62a3f --- /dev/null +++ b/docs/oracle.md @@ -0,0 +1,179 @@ +# Oracle Database Conversion Guide + +## Table of Contents +1. [Introduction](#introduction) +2. [Data Type Mappings](#data-type-mappings) +3. [Syntax Differences](#syntax-differences) +4. [Common Issues](#common-issues) +5. [Examples](#examples) + +## Introduction +This guide provides detailed information about converting Oracle database schemas to and from other database systems using SQLMapper. + +## Data Type Mappings + +### Oracle to MySQL +| Oracle Type | MySQL Type | Notes | +|------------|------------|-------| +| NUMBER(p,s) | DECIMAL(p,s) | For exact numeric values | +| NUMBER | BIGINT | When no precision specified | +| VARCHAR2 | VARCHAR | - | +| CLOB | LONGTEXT | - | +| BLOB | LONGBLOB | - | +| DATE | DATETIME | - | +| TIMESTAMP | TIMESTAMP | - | + +### Oracle to PostgreSQL +| Oracle Type | PostgreSQL Type | Notes | +|------------|-----------------|-------| +| NUMBER(p,s) | NUMERIC(p,s) | - | +| VARCHAR2 | VARCHAR | - | +| CLOB | TEXT | - | +| BLOB | BYTEA | - | +| DATE | TIMESTAMP | - | +| TIMESTAMP | TIMESTAMP | - | + +## Syntax Differences + +### Sequences +Oracle: +```sql +CREATE SEQUENCE my_sequence + START WITH 1 + INCREMENT BY 1 + NOCACHE + NOCYCLE; +``` + +MySQL equivalent: +```sql +CREATE TABLE my_sequence ( + id BIGINT NOT NULL AUTO_INCREMENT, + PRIMARY KEY (id) +); +``` + +PostgreSQL equivalent: +```sql +CREATE SEQUENCE my_sequence + START 1 + INCREMENT 1 + NO CYCLE; +``` + +### Stored Procedures +Oracle: +```sql +CREATE OR REPLACE PROCEDURE update_employee( + p_emp_id IN NUMBER, + p_salary IN NUMBER +) +IS +BEGIN + UPDATE employees + SET salary = p_salary + WHERE employee_id = p_emp_id; +END; +/ +``` + +MySQL equivalent: +```sql +DELIMITER // +CREATE PROCEDURE update_employee( + IN p_emp_id INT, + IN p_salary DECIMAL(10,2) +) +BEGIN + UPDATE employees + SET salary = p_salary + WHERE employee_id = p_emp_id; +END // +DELIMITER ; +``` + +## Common Issues + +### 1. Date Format Differences +Oracle's default date format differs from other databases. Use explicit format strings: +```sql +-- Oracle +TO_DATE('2023-12-27', 'YYYY-MM-DD') + +-- MySQL +STR_TO_DATE('2023-12-27', '%Y-%m-%d') + +-- PostgreSQL +TO_DATE('2023-12-27', 'YYYY-MM-DD') +``` + +### 2. Sequence Usage +When converting sequences, be aware that: +- MySQL doesn't support sequences natively +- PostgreSQL sequences require explicit nextval() calls +- Oracle sequences can be used in DEFAULT values + +### 3. NULL Handling +Oracle treats empty strings as NULL, while other databases distinguish between empty strings and NULL values. + +## Examples + +### Converting Table with Identity Column +```sql +-- Original Oracle Table +CREATE TABLE employees ( + emp_id NUMBER GENERATED ALWAYS AS IDENTITY, + name VARCHAR2(100), + salary NUMBER(10,2), + hire_date DATE, + CONSTRAINT pk_emp PRIMARY KEY (emp_id) +); + +-- MySQL Conversion +CREATE TABLE employees ( + emp_id BIGINT AUTO_INCREMENT, + name VARCHAR(100), + salary DECIMAL(10,2), + hire_date DATETIME, + PRIMARY KEY (emp_id) +); + +-- PostgreSQL Conversion +CREATE TABLE employees ( + emp_id SERIAL, + name VARCHAR(100), + salary NUMERIC(10,2), + hire_date TIMESTAMP, + PRIMARY KEY (emp_id) +); +``` + +### Converting Complex Types +```sql +-- Oracle Table with Complex Types +CREATE TABLE documents ( + doc_id NUMBER, + content CLOB, + metadata VARCHAR2(4000), + binary_data BLOB, + CONSTRAINT pk_doc PRIMARY KEY (doc_id) +); + +-- MySQL Conversion +CREATE TABLE documents ( + doc_id BIGINT, + content LONGTEXT, + metadata JSON, + binary_data LONGBLOB, + PRIMARY KEY (doc_id) +); + +-- PostgreSQL Conversion +CREATE TABLE documents ( + doc_id BIGINT, + content TEXT, + metadata JSONB, + binary_data BYTEA, + PRIMARY KEY (doc_id) +); +``` \ No newline at end of file diff --git a/docs/sqlserver.md b/docs/sqlserver.md new file mode 100644 index 0000000..3eee485 --- /dev/null +++ b/docs/sqlserver.md @@ -0,0 +1,200 @@ +# SQL Server Database Conversion Guide + +## Table of Contents +1. [Introduction](#introduction) +2. [Data Type Mappings](#data-type-mappings) +3. [Syntax Differences](#syntax-differences) +4. [Common Issues](#common-issues) +5. [Examples](#examples) + +## Introduction +This guide provides detailed information about converting SQL Server database schemas to and from other database systems using SQLMapper. + +## Data Type Mappings + +### SQL Server to MySQL +| SQL Server Type | MySQL Type | Notes | +|----------------|------------|-------| +| BIGINT | BIGINT | - | +| INT | INT | - | +| SMALLINT | SMALLINT | - | +| TINYINT | TINYINT | Range differences | +| DECIMAL(p,s) | DECIMAL(p,s) | - | +| VARCHAR(n) | VARCHAR(n) | - | +| NVARCHAR(n) | VARCHAR(n) | UTF-8 encoding | +| TEXT | LONGTEXT | - | +| DATETIME2 | DATETIME | Precision differences | +| UNIQUEIDENTIFIER | CHAR(36) | - | + +### SQL Server to PostgreSQL +| SQL Server Type | PostgreSQL Type | Notes | +|----------------|-----------------|-------| +| BIGINT | BIGINT | - | +| INT | INTEGER | - | +| SMALLINT | SMALLINT | - | +| DECIMAL(p,s) | NUMERIC(p,s) | - | +| VARCHAR(n) | VARCHAR(n) | - | +| NVARCHAR(n) | VARCHAR(n) | - | +| TEXT | TEXT | - | +| DATETIME2 | TIMESTAMP | - | +| UNIQUEIDENTIFIER | UUID | - | + +## Syntax Differences + +### Identity Columns +SQL Server: +```sql +CREATE TABLE users ( + id INT IDENTITY(1,1) PRIMARY KEY, + username VARCHAR(50) +); +``` + +MySQL equivalent: +```sql +CREATE TABLE users ( + id INT AUTO_INCREMENT PRIMARY KEY, + username VARCHAR(50) +); +``` + +PostgreSQL equivalent: +```sql +CREATE TABLE users ( + id SERIAL PRIMARY KEY, + username VARCHAR(50) +); +``` + +### Stored Procedures +SQL Server: +```sql +CREATE PROCEDURE UpdateEmployee + @EmpID INT, + @Salary DECIMAL(10,2) +AS +BEGIN + UPDATE Employees + SET Salary = @Salary + WHERE EmployeeID = @EmpID; +END; +``` + +MySQL equivalent: +```sql +DELIMITER // +CREATE PROCEDURE UpdateEmployee( + IN p_EmpID INT, + IN p_Salary DECIMAL(10,2) +) +BEGIN + UPDATE Employees + SET Salary = p_Salary + WHERE EmployeeID = p_EmpID; +END // +DELIMITER ; +``` + +## Common Issues + +### 1. Collation Differences +SQL Server uses different collation naming conventions: +```sql +-- SQL Server +COLLATE SQL_Latin1_General_CP1_CI_AS + +-- MySQL +COLLATE utf8mb4_general_ci + +-- PostgreSQL +COLLATE "en_US.utf8" +``` + +### 2. DateTime Handling +```sql +-- SQL Server +GETDATE() +DATEADD(day, 1, GETDATE()) + +-- MySQL +NOW() +DATE_ADD(NOW(), INTERVAL 1 DAY) + +-- PostgreSQL +CURRENT_TIMESTAMP +CURRENT_TIMESTAMP + INTERVAL '1 day' +``` + +### 3. String Concatenation +```sql +-- SQL Server +SELECT FirstName + ' ' + LastName + +-- MySQL +SELECT CONCAT(FirstName, ' ', LastName) + +-- PostgreSQL +SELECT FirstName || ' ' || LastName +``` + +## Examples + +### Converting Table with Computed Columns +```sql +-- Original SQL Server Table +CREATE TABLE orders ( + order_id INT IDENTITY(1,1), + quantity INT, + unit_price DECIMAL(10,2), + total_price AS (quantity * unit_price), + CONSTRAINT pk_orders PRIMARY KEY (order_id) +); + +-- MySQL Conversion +CREATE TABLE orders ( + order_id INT AUTO_INCREMENT, + quantity INT, + unit_price DECIMAL(10,2), + total_price DECIMAL(10,2) GENERATED ALWAYS AS (quantity * unit_price) STORED, + PRIMARY KEY (order_id) +); + +-- PostgreSQL Conversion +CREATE TABLE orders ( + order_id SERIAL, + quantity INT, + unit_price NUMERIC(10,2), + total_price NUMERIC(10,2) GENERATED ALWAYS AS (quantity * unit_price) STORED, + PRIMARY KEY (order_id) +); +``` + +### Converting Table with Custom Types +```sql +-- SQL Server Table with Custom Types +CREATE TABLE customer_data ( + id INT IDENTITY(1,1), + customer_code UNIQUEIDENTIFIER DEFAULT NEWID(), + status VARCHAR(20), + metadata NVARCHAR(MAX), + CONSTRAINT pk_customer PRIMARY KEY (id) +); + +-- MySQL Conversion +CREATE TABLE customer_data ( + id INT AUTO_INCREMENT, + customer_code CHAR(36) DEFAULT (UUID()), + status VARCHAR(20), + metadata JSON, + PRIMARY KEY (id) +); + +-- PostgreSQL Conversion +CREATE TABLE customer_data ( + id SERIAL, + customer_code UUID DEFAULT gen_random_uuid(), + status VARCHAR(20), + metadata JSONB, + PRIMARY KEY (id) +); +``` \ No newline at end of file diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 67feb76..e58df22 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,151 +1,227 @@ -# Troubleshooting Guide - -## Common Issues - -### 1. Conversion Errors - -#### Unsupported Data Types -**Problem:** Some data types from the source database are not supported in the target database. - -**Solution:** -- Check the data type mapping table -- Use alternative data types -- Add custom conversion functions - -#### Syntax Errors -**Problem:** SQL statements cannot be parsed or are invalid. - -**Solution:** -- Check SQL statement syntax -- Verify database version compatibility -- Simplify complex expressions - -### 2. Performance Issues - -#### Large Schema Conversions -**Problem:** Converting large database schemas is slow. - -**Solution:** -- Break schema into smaller parts -- Remove unnecessary tables and fields -- Optimize memory usage - -#### Index Issues -**Problem:** Indexes are not converting properly or causing performance issues. - -**Solution:** -- Check index type compatibility -- Remove unnecessary indexes -- Optimize index creation order - -### 3. Data Integrity - -#### Foreign Key Constraints -**Problem:** Foreign key constraints are not working properly. - -**Solution:** -- Check referential integrity -- Adjust table creation order -- Temporarily disable constraints - -#### Character Set Issues -**Problem:** Character encoding issues in text data. - -**Solution:** -- Check character set and collation settings -- Prefer UTF-8 usage -- Escape special characters - -### 4. Database-Specific Issues - -#### MySQL to PostgreSQL -**Problem:** -- AUTO_INCREMENT behavior differs -- ON UPDATE CURRENT_TIMESTAMP not supported -- UNSIGNED data types not available - -**Solution:** -- Use SERIAL or IDENTITY -- Implement timestamp updates with triggers -- Convert data types appropriately - -#### PostgreSQL to SQLite -**Problem:** -- Complex data types not supported -- Schema changes are limited -- Advanced index types not available - -**Solution:** -- Convert to simple data types -- Manage schema changes manually -- Use alternative indexing strategies - -#### SQLite to Oracle -**Problem:** -- Auto-incrementing fields work differently -- Data type incompatibilities -- Trigger syntax differences - -**Solution:** -- Use SEQUENCE and TRIGGER -- Map data types to Oracle equivalents -- Rewrite triggers - -## Best Practices - -1. **Testing** - - Test with small dataset first - - Verify all CRUD operations - - Perform performance tests - -2. **Backup** - - Take backup before conversion - - Use incremental backup strategy - - Prepare rollback plan - -3. **Documentation** - - Document changes - - Note known issues - - Share solutions - -4. **Monitoring** - - Check error logs - - Monitor performance metrics - - Evaluate user feedback - -## Frequently Asked Questions - -### General Questions - -**Q: Which database versions does SQLMapper support?** -A: Supports latest stable versions. Check documentation for detailed version compatibility. - -**Q: Will there be data loss during conversion?** -A: Data loss can be prevented with proper configuration and testing. Always take backups. - -**Q: Is it suitable for large databases?** -A: Yes, but may require chunked conversion and optimization. - -### Technical Questions - -**Q: How are custom data types handled?** -A: You can add custom conversion functions or use default mappings. - -**Q: Are triggers and stored procedures converted?** -A: Simple triggers are converted automatically, complex ones require manual intervention. - -**Q: How are schema changes managed?** -A: Change scripts are generated and applied in sequence. - -## Contact and Support - -- GitHub Issues: Bug reports and feature requests -- Documentation: Detailed usage guides -- Community: Discussion forums and contributions - -## Version History - -- v1.0.0: Initial stable release -- v1.1.0: Performance improvements -- v1.2.0: New database support -- v1.3.0: Bug fixes and optimizations \ No newline at end of file +# SQLMapper Troubleshooting Guide + +## Table of Contents +1. [Common Error Messages](#common-error-messages) +2. [Database-Specific Issues](#database-specific-issues) +3. [Performance Issues](#performance-issues) +4. [Conversion Problems](#conversion-problems) +5. [CLI Issues](#cli-issues) + +## Common Error Messages + +### "Invalid SQL Syntax" +**Problem**: SQLMapper fails to parse the input SQL file. +**Solution**: +1. Verify the source database type is correctly detected +2. Check for unsupported SQL features +3. Remove any database-specific comments or directives +4. Split complex statements into simpler ones + +### "Unsupported Data Type" +**Problem**: Encountered a data type that doesn't have a direct mapping. +**Solution**: +1. Check the data type mapping documentation +2. Use the `--force-type-mapping` flag with a custom mapping +3. Modify the source SQL to use a supported type +4. Implement a custom type converter + +### "File Access Error" +**Problem**: Cannot read input file or write output file. +**Solution**: +1. Verify file permissions +2. Check if the file path is correct +3. Ensure sufficient disk space +4. Run the command with appropriate privileges + +## Database-Specific Issues + +### MySQL Issues + +#### Character Set Conversion +**Problem**: UTF8MB4 character set conversion fails. +**Solution**: +```sql +-- Original MySQL +ALTER TABLE users CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; + +-- Modified for PostgreSQL +-- Add this before table creation: +SET client_encoding = 'UTF8'; +``` + +#### JSON Column Conversion +**Problem**: JSON columns not properly converted. +**Solution**: +```sql +-- MySQL +ALTER TABLE data MODIFY COLUMN json_col JSON; + +-- PostgreSQL equivalent +ALTER TABLE data ALTER COLUMN json_col TYPE JSONB USING json_col::JSONB; + +-- SQLite equivalent (stored as TEXT) +ALTER TABLE data ALTER COLUMN json_col TEXT; +``` + +### PostgreSQL Issues + +#### Array Type Conversion +**Problem**: Array types not supported in target database. +**Solution**: +```sql +-- PostgreSQL source +CREATE TABLE items ( + id SERIAL, + tags TEXT[] +); + +-- MySQL conversion +CREATE TABLE items ( + id INT AUTO_INCREMENT, + tags JSON, + PRIMARY KEY (id) +); + +-- Add migration script: +INSERT INTO items_new (id, tags) +SELECT id, JSON_ARRAY(UNNEST(tags)) FROM items; +``` + +### Oracle Issues + +#### ROWID Handling +**Problem**: ROWID pseudo-column not supported. +**Solution**: +```sql +-- Oracle source +SELECT * FROM employees WHERE ROWID = 'AAASuZAABAAALvVAAA'; + +-- MySQL conversion +ALTER TABLE employees ADD COLUMN row_identifier BIGINT AUTO_INCREMENT; + +-- PostgreSQL conversion +ALTER TABLE employees ADD COLUMN row_identifier BIGSERIAL; +``` + +## Performance Issues + +### Large File Processing + +**Problem**: Memory usage spikes with large SQL files. +**Solution**: +1. Use the `--chunk-size` flag to process in smaller batches +2. Enable streaming mode with `--stream` +3. Split large files into smaller ones +4. Use the `--optimize-memory` flag + +Example: +```bash +sqlmapper --file=large_dump.sql --to=mysql --chunk-size=1000 --optimize-memory +``` + +### Slow Conversion Speed + +**Problem**: Conversion takes longer than expected. +**Solution**: +1. Enable parallel processing: +```bash +sqlmapper --file=dump.sql --to=postgres --parallel-workers=4 +``` + +2. Optimize input SQL: +```sql +-- Before +SELECT * FROM large_table WHERE complex_condition; + +-- After +SELECT needed_columns FROM large_table WHERE simple_condition; +``` + +## Conversion Problems + +### Data Loss Prevention + +**Problem**: Potential data truncation during conversion. +**Solution**: +1. Use the `--strict` flag to fail on potential data loss +2. Add data validation queries: + +```sql +-- Check for oversized data +SELECT column_name, MAX(LENGTH(column_name)) as max_length +FROM table_name +GROUP BY column_name +HAVING max_length > new_size_limit; +``` + +### Constraint Violations + +**Problem**: Foreign key constraints fail after conversion. +**Solution**: +1. Use `--defer-constraints` flag +2. Add this to your conversion: +```sql +-- At the start +SET FOREIGN_KEY_CHECKS = 0; -- MySQL +SET CONSTRAINTS ALL DEFERRED; -- PostgreSQL + +-- At the end +SET FOREIGN_KEY_CHECKS = 1; -- MySQL +SET CONSTRAINTS ALL IMMEDIATE; -- PostgreSQL +``` + +## CLI Issues + +### Command Line Arguments + +**Problem**: Incorrect argument format +**Solution**: +```bash +# Incorrect +sqlmapper -file dump.sql -to mysql + +# Correct +sqlmapper --file=dump.sql --to=mysql +``` + +### Environment Setup + +**Problem**: Path or environment issues +**Solution**: +1. Add to PATH: +```bash +export PATH=$PATH:/path/to/sqlmapper/bin +``` + +2. Set required environment variables: +```bash +export SQLMAPPER_HOME=/path/to/sqlmapper +export SQLMAPPER_CONFIG=/path/to/config.json +``` + +### Debug Mode + +When encountering unexpected issues: +```bash +sqlmapper --file=dump.sql --to=mysql --debug --verbose +``` + +This will provide: +- Detailed error messages +- Stack traces +- SQL parsing steps +- Conversion process logs + +### Common Flags Reference + +```bash +--file= # Input SQL file +--to= # Target database type +--debug # Enable debug mode +--verbose # Detailed output +--strict # Strict conversion mode +--force # Ignore non-critical errors +--dry-run # Preview conversion +--config= # Custom config file +--output= # Output file path +``` \ No newline at end of file