Fixed width support. #391
Open
etspaceman wants to merge 2 commits into
Open
Conversation
…to avoid duplicated code. Largely derived from https://github.com/quartethealth/spark-csv
|
Nice job |
Member
|
If I understood correctly, this library now avoids to add new features but only accepts critical bug fixes. How about proposing this into Spark? |
Author
|
I hadn't realized that. This is disappointing. I'll see if I can get this proposed to the spark readers though. |
|
Hi, Is fixed width csv still not supported in spark / databricks for csv parsing? thanks |
Member
|
@kurtmaile this repository is on maintenance mode and does not accept new features. All new feature development should be directed to Apache Spark CSV data source implementation. |
|
Sure thanks.
So with regards to my general question on csv parsing in spark, do you know
of a spark fixed width csv parser feature in the spark data soruce
implementation?
Cheers
…On 14 June 2018 at 02:27, Hossein Falaki ***@***.***> wrote:
@kurtmaile <https://github.com/kurtmaile> this repository is on
maintenance mode and does not accept new features. All new feature
development should be directed to Apache Spark CSV data source
implementation.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#391 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ATIvhXJPPFY-BqfoNiqezUBMmzB9sZWdks5t8bv-gaJpZM4Ki7M4>
.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Relation, LineReader and BulkReader traits to avoid duplicated code. Largely derived from https://github.com/quartethealth/spark-csv and https://github.com/quartethealth/spark-fixedwidth.
This is in response to the following PR (created by @blrnw3) being closed without a merge:
#259
A fixed-width parser is a very common use case that I think several users would enjoy using. We plan to use this in our current production environment.