재미있는 CSV를 구문 분석하는 정규 표현식?

Question

AWK를 사용하여 CSV 파일을 구문 분석해야합니다. CSV에서 한 라인은 다음과 같이 수 :

"hello, world?",1 thousand,"oneword",,,"last one" 

몇 가지 중요한 관찰 : -unquoted 필드가 여러 세계가 될 수 쉼표와 여러 단어를 포함 할 수 있습니다 인용 문자열 내부 -field는 -field 단지함으로써 비어있을 수 있습니다 행의 두 쉼표

이 줄을 올바르게 나눌 수있는 정규식을 작성하는 데 대한 단서가 있습니까?

감사합니다.

Answer 1

많은 사람들이 관찰했듯이 CSV는 처음 나타나는 것보다 더 어려운 형식입니다. 많은 경우와 애매 모호함이 있습니다. 귀하의 예에서 모호한 예로서 ',,,'은 쉼표 또는 두 개의 빈 필드가있는 필드입니까?

CSV를 잘 처리 할 수있는 Perl, python, Java 등은 라이브러리를 잘 테스트했기 때문에 CSV를 처리하는 것이 좋습니다. 정규식은 더 약해집니다.

AWK를 사용하면 AWK 기능이 THIS과 함께 성공했습니다. AWK, gawk 및 nawk에서 작동합니다.

#!/usr/bin/awk -f 
#************************************************************************** 
# 
# This file is in the public domain. 
# 
# For more information email LoranceStinson+csv@gmail.com. 
# Or see http://lorance.freeshell.org/csv/ 
# 
# Parse a CSV string into an array. 
# The number of fields found is returned. 
# In the event of an error a negative value is returned and csverr is set to 
# the error. See below for the error values. 
# 
# Parameters: 
# string = The string to parse. 
# csv  = The array to parse the fields into. 
# sep  = The field separator character. Normally , 
# quote = The string quote character. Normally " 
# escape = The quote escape character. Normally " 
# newline = Handle embedded newlines. Provide either a newline or the 
#   string to use in place of a newline. If left empty embedded 
#   newlines cause an error. 
# trim = When true spaces around the separator are removed. 
#   This affects parsing. Without this a space between the 
#   separator and quote result in the quote being ignored. 
# 
# These variables are private: 
# fields = The number of fields found thus far. 
# pos  = Where to pull a field from the string. 
# strtrim = True when a string is found so we know to remove the quotes. 
# 
# Error conditions: 
# -1 = Unable to read the next line. 
# -2 = Missing end quote. 
# -3 = Missing separator. 
# 
# Notes: 
# The code assumes that every field is preceded by a separator, even the 
# first field. This makes the logic much simpler, but also requires a 
# separator be prepended to the string before parsing. 
#************************************************************************** 
function parse_csv(string,csv,sep,quote,escape,newline,trim, fields,pos,strtrim) { 
    # Make sure there is something to parse. 
    if (length(string) == 0) return 0; 
    string = sep string; # The code below assumes ,FIELD. 
    fields = 0; # The number of fields found thus far. 
    while (length(string) > 0) { 
     # Remove spaces after the separator if requested. 
     if (trim && substr(string, 2, 1) == " ") { 
      if (length(string) == 1) return fields; 
      string = substr(string, 2); 
      continue; 
     } 
     strtrim = 0; # Used to trim quotes off strings. 
     # Handle a quoted field. 
     if (substr(string, 2, 1) == quote) { 
      pos = 2; 
      do { 
       pos++ 
       if (pos != length(string) && 
        substr(string, pos, 1) == escape && 
        (substr(string, pos + 1, 1) == quote || 
        substr(string, pos + 1, 1) == escape)) { 
        # Remove escaped quote characters. 
        string = substr(string, 1, pos - 1) substr(string, pos + 1); 
       } else if (substr(string, pos, 1) == quote) { 
        # Found the end of the string. 
        strtrim = 1; 
       } else if (newline && pos >= length(string)) { 
        # Handle embedded newlines if requested. 
        if (getline == -1) { 
         csverr = "Unable to read the next line."; 
         return -1; 
        } 
        string = string newline $0; 
       } 
      } while (pos < length(string) && strtrim == 0) 
      if (strtrim == 0) { 
       csverr = "Missing end quote."; 
       return -2; 
      } 
     } else { 
      # Handle an empty field. 
      if (length(string) == 1 || substr(string, 2, 1) == sep) { 
       csv[fields] = ""; 
       fields++; 
       if (length(string) == 1) 
        return fields; 
       string = substr(string, 2); 
       continue; 
      } 
      # Search for a separator. 
      pos = index(substr(string, 2), sep); 
      # If there is no separator the rest of the string is a field. 
      if (pos == 0) { 
       csv[fields] = substr(string, 2); 
       fields++; 
       return fields; 
      } 
     } 
     # Remove spaces after the separator if requested. 
     if (trim && pos != length(string) && substr(string, pos + strtrim, 1) == " ") { 
      trim = strtrim 
      # Count the number fo spaces found. 
      while (pos < length(string) && substr(string, pos + trim, 1) == " ") { 
       trim++ 
      } 
      # Remove them from the string. 
      string = substr(string, 1, pos + strtrim - 1) substr(string, pos + trim); 
      # Adjust pos with the trimmed spaces if a quotes string was not found. 
      if (!strtrim) { 
       pos -= trim; 
      } 
     } 
     # Make sure we are at the end of the string or there is a separator. 
     if ((pos != length(string) && substr(string, pos + 1, 1) != sep)) { 
      csverr = "Missing separator."; 
      return -3; 
     } 
     # Gather the field. 
     csv[fields] = substr(string, 2 + strtrim, pos - (1 + strtrim * 2)); 
     fields++; 
     # Remove the field from the string for the next pass. 
     string = substr(string, pos + 1); 
    } 
    return fields; 
} 

{ 
    num_fields = parse_csv($0, csv, ",", "\"", "\"", "\\n", 1); 
    if (num_fields < 0) { 
     printf "ERROR: %s (%d) -> %s\n", csverr, num_fields, $0; 
    } else { 
     printf "%s -> \n", $0; 
     printf "%s fields\n", num_fields; 
     for (i = 0;i < num_fields;i++) { 
      printf "%s\n", csv[i]; 
     } 
     printf "|\n"; 
    } 
} 

귀하의 예제 데이터를 실행하면 생성합니다

"hello, world?",1 thousand,"oneword",,,"last one" -> 
6 fields 
hello, world? 
1 thousand 
oneword 


last one 
| 

예 펄 솔루션 :

$ echo '"hello, world?",1 thousand,"oneword",,,"last one"' | 
perl -lnE 'for(/(?:^|,)("(?:[^"]+|"")*"|[^,]*)/g) { s/"$//; s/""/"/g if (s/^"//); 
say}' 

Answer 2

이 시도 :

그래도 난 AWK와 함께 테스트하지 않았습니다

^(("(?:[^"]|"")*"|[^,]*)(,("(?:[^"]|"")*"|[^,]*))*)$ 

.