Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Xml deserialization #37

Merged
merged 16 commits into from
Oct 27, 2019
Merged

Xml deserialization #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
47c1864
Initial working prototype
Blacksmoke16 Sep 1, 2019
e322541
Fix lint issue
Blacksmoke16 Sep 1, 2019
2cbe8fe
Refactor to better handle arrays
Blacksmoke16 Sep 1, 2019
7f9b49e
Support empty elements
Blacksmoke16 Sep 15, 2019
b6327aa
fix attributes and inline arrays within arrays
Blacksmoke16 Sep 17, 2019
04e3801
Fix lint issue
Blacksmoke16 Oct 9, 2019
616104e
better handle colorized JSON output
Blacksmoke16 Oct 9, 2019
e51a71b
Merge branches 'master' and 'master' of https://github.com/Blacksmoke…
Blacksmoke16 Oct 9, 2019
3470c43
Support CDATA elements
Blacksmoke16 Oct 20, 2019
c0751c8
Fix spec
Blacksmoke16 Oct 20, 2019
652764d
Greatly simplify XML deserialization logic
Blacksmoke16 Oct 20, 2019
83d9c08
No longer reopen XML::Node
Blacksmoke16 Oct 20, 2019
4990ce0
Remove leftover testing file
Blacksmoke16 Oct 21, 2019
1f25ac3
Some final cleanup
Blacksmoke16 Oct 21, 2019
814b7b6
Add some examples to the README
Blacksmoke16 Oct 22, 2019
ded9133
Update crystal install link
Blacksmoke16 Oct 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 38 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ A performant, portable [jq](https://github.com/stedolan/jq/) wrapper thats facil

* Compiles to a single binary for easy portability.
* Performant, similar performance with JSON data compared to `jq`. Slightly longer execution time when going to/from a non-JSON format.
* Supports XML and YAML as additional output formats.
* Supports XML and YAML as additional input/output formats.

## Installation

Expand All @@ -32,7 +32,7 @@ brew install oq

If building from source, `jq` will need to be installed separately. Installation instructions can be found in the [official documentation](https://stedolan.github.io/jq/).

Requires Crystal to be installed, see the [installation documentation](https://crystal-lang.org/reference/installation/).
Requires Crystal to be installed, see the [installation documentation](https://crystal-lang.org/install).

```bash
git clone https://github.com/Blacksmoke16/oq.git
Expand All @@ -48,6 +48,42 @@ The built binary will be available as `./bin/oq`. This can be relocated elsewhe

Use the `oq` binary, with a few optional custom arguments, see `oq --help`. All other arguments get passed to `jq`. See [jq manual](https://stedolan.github.io/jq/manual/) for details.

### Examples

#### Consume JSON and output XML

```bash
echo '{"name": "Jim"}' | oq -o xml .
<?xml version="1.0" encoding="UTF-8"?>
<root>
<name>Jim</name>
</root>
```

#### Consume YAML from a file and output JSON

data.yaml

```yaml
---
name: Jim
numbers:
- 1
- 2
- 3
```

```bash
oq -i yaml -o xml . data.yaml
<?xml version="1.0" encoding="UTF-8"?>
<root>
<name>Jim</name>
<numbers>1</numbers>
<numbers>2</numbers>
<numbers>3</numbers>
</root>
```

## Roadmap

Plans for `1.0.0`:
Expand Down
304 changes: 303 additions & 1 deletion spec/converters/xml_spec.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,309 @@
require "../spec_helper"

WITH_WHITESPACE = <<-XML
<item>
<flagID>0</flagID>
<itemID>0</itemID>
<locationID>0</locationID>
<ownerID>0</ownerID>
<quantity>-1</quantity>
<typeID>0</typeID>
</item>
XML

XML_SCALAR_ARRAY = <<-XML
<?xml version="1.0" encoding="utf-8"?>
<items>
<number>1</number>
<number>2</number>
<number>3</number>
</items>
XML

XML_CDATA = <<-XML
<desc><![CDATA[<message>Some Description</message>]]></desc>
XML

XML_OBJECT_ARRAY = <<-XML
<?xml version="1.0" encoding="utf-8"?>
<items>
<item>
<flagID>0</flagID>
<itemID>0</itemID>
<locationID>0</locationID>
<ownerID>0</ownerID>
<quantity>-1</quantity>
<typeID>0</typeID>
</item>
<item>
<flagID>0</flagID>
<itemID>1</itemID>
<locationID>0</locationID>
<ownerID>0</ownerID>
<quantity>-1</quantity>
<typeID>0</typeID>
</item>
</items>
XML

XML_NESTED_OBJECT_ARRAY = <<-XML
<?xml version='1.0' ?>
<!DOCTYPE root SYSTEM "http://www.cs.washington.edu/research/projects/xmltk/xmldata/data/auctions/ebay.dtd">
<root>
<listing>
<seller_info>
<seller_name> cubsfantony</seller_name>
<seller_rating> 848</seller_rating>
</seller_info>
<payment_types>Visa/MasterCard, Money Order/Cashiers Checks, Personal Checks, See item description for payment methods accepted</payment_types>
</listing>
<listing>
<seller_info>
<seller_name> ct-inc</seller_name>
<seller_rating> 403</seller_rating>
</seller_info>
<payment_types>Visa/MasterCard, Discover, Money Order/Cashiers Checks, Personal Checks, See item description for payment methods accepted</payment_types>
</listing>
</root>
XML

XML_INLINE_ARRAY = <<-XML
<article key="tr/ibm/RJ2144">
<author>E. F. Codd</author>
<author>Robert S. Arnold</author>
<author>Jean-Marc Cadiou</author>
<author>Chin-Liang Chang</author>
<author>Nick Roussopoulos</author>
<title>RENDEZVOUS Version 1: An Experimental English Language Query Formulation System for Casual Users of Relational Data Bases.</title>
<journal>IBM Research Report</journal>
<volume>RJ2144</volume>
<month>January</month>
<year>1978</year>
<ee>db/labs/ibm/RJ2144.html</ee>
<cdrom>ibmTR/rj2144.pdf</cdrom>
</article>
XML

XML_INLINE_ARRAY_WITHIN_ARRAY = <<-XML
<articles>
<article key="tr/dec/SRC1997-018">
<year>1997</year>
<ee>db/labs/dec/SRC1997-018.html</ee>
<ee>http://www.mcjones.org/System_R/SQL_Reunion_95/</ee>
</article>
<article key="tr/gte/TR-0263-08-94-165">
<ee>db/labs/gte/TR-0263-08-94-165.html</ee>
<year>1994</year>
</article>
</articles>
XML

XML_DOCTYPE = <<-XML
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE dblp SYSTEM "dblp.dtd">
<dblp>
<mastersthesis key="ms/Brown92">
<author>Kurt P. Brown</author>
<title>PRPL: A Database Workload Specification Language, v1.3.</title>
<year>1992</year>
<school>Univ. of Wisconsin-Madison</school>
</mastersthesis>
</dblp>
XML

XML_ATTRIBUTE_IN_ARRAY = <<-XML
<jobs>
<ad>
<salary currency="CAD">80000</salary>
<working_hours>full-time</working_hours>
</ad>
<ad>
<working_hours>full-time</working_hours>
</ad>
</jobs>
XML

XML_ATTRIBUTE_IN_ARRAY_ROOT_ELEMENT = <<-XML
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE dblp SYSTEM "dblp.dtd">
<dblp>
<mastersthesis key="ms/Brown92">
<author>Kurt P. Brown</author>
</mastersthesis>
<mastersthesis key="ms/Yurek97">
<author>Tolga Yurek</author>
</mastersthesis>
</dblp>
XML

XML_ALL_EMPTY = <<-XML
<root>
<one> </one>
<two>
</two>
</root>
XML

describe OQ::Converters::Xml do
pending ".deserialize" do
describe ".deserialize" do
describe "should raise if invalid" do
it "should output correctly" do
run_binary(%(<root id="1<child/></root>), args: ["-i", "xml", "-c", "."]) do |_, status, error|
error.should eq "oq error: Couldn't find end of Start Tag root\n"
status.exit_code.should eq 1
end
end
end

describe Object do
describe "a key/value pair" do
it "should output correctly" do
run_binary(%(<person>Fred</person>), args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"person":"Fred"}\n)
end
end
end

describe "that has only empty children elements" do
it "should output an object with null values" do
run_binary(XML_ALL_EMPTY, args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"root":{"one":null,"two":null}}\n)
end
end
end

describe "with whitespace" do
it "should output correctly" do
run_binary(WITH_WHITESPACE, args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"item":{"flagID":"0","itemID":"0","locationID":"0","ownerID":"0","quantity":"-1","typeID":"0"}}\n)
end
end
end

describe "with the prolog" do
it "should output correctly" do
run_binary(%(<?xml version="1.0" encoding="utf-8"?><item><typeID>0</typeID></item>), args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"item":{"typeID":"0"}}\n)
end
end
end

describe "a simple object" do
it "should output correctly" do
run_binary(%(<person><firstname>Jane</firstname><lastname>Doe</lastname></person>), args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"person":{"firstname":"Jane","lastname":"Doe"}}\n)
end
end
end

describe "attributes" do
it "should output correctly" do
run_binary(%(<person id="1" foo="bar"><firstname>Jane</firstname><lastname>Doe</lastname></person>), args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"person":{"@id":"1","@foo":"bar","firstname":"Jane","lastname":"Doe"}}\n)
end
end
end

describe "nested objects" do
it "should output correctly" do
run_binary(%(<person><firstname>Jane</firstname><lastname>Doe</lastname><location><zip>15061</zip><address>123 Foo Street</address></location></person>), args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"person":{"firstname":"Jane","lastname":"Doe","location":{"zip":"15061","address":"123 Foo Street"}}}\n)
end
end
end

describe "complex object" do
it "should output correctly" do
run_binary(%(<root><x a="1"><a>2</a></x><y b="3">4</y></root>), args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"root":{"x":{"@a":"1","a":"2"},"y":{"@b":"3","#text":"4"}}}\n)
end
end
end

describe "with mixed content" do
it "should output correctly" do
run_binary(%(<root>x<y>z</y></root>), args: ["-i", "xml", "-c", ".root"]) do |output|
output.should eq %({"y":"z"}\n)
end
end
end

describe "with an inline array" do
it "should output correctly" do
run_binary(XML_INLINE_ARRAY, args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"article":{"@key":"tr/ibm/RJ2144","author":["E. F. Codd","Robert S. Arnold","Jean-Marc Cadiou","Chin-Liang Chang","Nick Roussopoulos"],"title":"RENDEZVOUS Version 1: An Experimental English Language Query Formulation System for Casual Users of Relational Data Bases.","journal":"IBM Research Report","volume":"RJ2144","month":"January","year":"1978","ee":"db/labs/ibm/RJ2144.html","cdrom":"ibmTR/rj2144.pdf"}}\n)
end
end
end

describe "with a doctype" do
it "should output correctly" do
run_binary(XML_DOCTYPE, args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"dblp":{"mastersthesis":{"@key":"ms/Brown92","author":"Kurt P. Brown","title":"PRPL: A Database Workload Specification Language, v1.3.","year":"1992","school":"Univ. of Wisconsin-Madison"}}}\n)
end
end
end

describe "with CDATA" do
it "should output correctly" do
run_binary(XML_CDATA, args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"desc":"<message>Some Description</message>"}\n)
end
end
end
end

describe Array do
describe "of scalar values" do
it "should output correctly" do
run_binary(XML_SCALAR_ARRAY, args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"items":{"number":["1","2","3"]}}\n)
end
end
end

describe "of objects" do
describe "with no nested objects" do
it "should output correctly" do
run_binary(XML_OBJECT_ARRAY, args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"items":{"item":[{"flagID":"0","itemID":"0","locationID":"0","ownerID":"0","quantity":"-1","typeID":"0"},{"flagID":"0","itemID":"1","locationID":"0","ownerID":"0","quantity":"-1","typeID":"0"}]}}\n)
end
end
end

describe "with an inline array" do
it "should output correctly" do
run_binary(XML_INLINE_ARRAY_WITHIN_ARRAY, args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"articles":{"article":[{"@key":"tr/dec/SRC1997-018","year":"1997","ee":["db/labs/dec/SRC1997-018.html","http://www.mcjones.org/System_R/SQL_Reunion_95/"]},{"@key":"tr/gte/TR-0263-08-94-165","ee":"db/labs/gte/TR-0263-08-94-165.html","year":"1994"}]}}\n)
end
end
end

describe "with nested objects" do
it "should output correctly" do
run_binary(XML_NESTED_OBJECT_ARRAY, args: ["-i", "xml", "-c", ".root.listing"]) do |output|
output.should eq %([{"seller_info":{"seller_name":" cubsfantony","seller_rating":" 848"},"payment_types":"Visa/MasterCard, Money Order/Cashiers Checks, Personal Checks, See item description for payment methods accepted"},{"seller_info":{"seller_name":" ct-inc","seller_rating":" 403"},"payment_types":"Visa/MasterCard, Discover, Money Order/Cashiers Checks, Personal Checks, See item description for payment methods accepted"}]\n)
end
end
end
end

describe "with object that has an attribute" do
it "should output correctly" do
run_binary(XML_ATTRIBUTE_IN_ARRAY, args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"jobs":{"ad":[{"salary":{"@currency":"CAD","#text":"80000"},"working_hours":"full-time"},{"working_hours":"full-time"}]}}\n)
end
end
end

describe "where array object element has an attribute" do
it "should output correctly" do
run_binary(XML_ATTRIBUTE_IN_ARRAY_ROOT_ELEMENT, args: ["-i", "xml", "-c", "."]) do |output|
output.should eq %({"dblp":{"mastersthesis":[{"@key":"ms/Brown92","author":"Kurt P. Brown"},{"@key":"ms/Yurek97","author":"Tolga Yurek"}]}}\n)
end
end
end
end
end

describe ".serialize" do
Expand Down
Loading